You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "Shields, Paul Michael" <pa...@hpe.com> on 2023/12/08 21:45:15 UTC
Trouble with Replication HA Master/Slave config performing failback
Hi,
I am having an issue with failback to the master broker. I start the master broker then the slave broker. I see the replication from the master to the slave happen. I stop the master/active broker and a failover happens to the slave as expected. When the master is started a failback to it is expected but does not happen. The following is my ha-policy config. Did I miss something in the ha-policy configuration? We are using version 2.28.0 and java17.
Thanks,
Paul
Master broker.xml config
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
Slave broker.xml config
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
The following are the logs from the master and slave brokers
HA “master” broker log after restart
2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for protocols [MQTT]
2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for protocols [CORE]
2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0 [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003: Starting embedded web server
2023-12-08 20:42:57,625 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message with ID: 15, result: NO_BINDINGS
2023-12-08 20:42:57,627 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created queue: QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@15525ed3
2023-12-08 20:42:57,657 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl [ID=47edf16f, clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2[ID=47edf16f<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2[ID=47edf16f>, local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
2023-12-08 20:42:57,720 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server] AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@15525ed3 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@15525ed3 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883[nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f, connector=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local, address=, server=ActiveMQServerImpl::name=amq-broker])) [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local], discoveryGroupConfiguration=null]] is connected
2023-12-08 20:42:57,755 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2023-12-08 20:42:57,766 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message with ID: 18, result: NO_BINDINGS
2023-12-08 20:42:57,767 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message with ID: 20, result: NO_BINDINGS
2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to Hawtio 2.15.0
2023-12-08 20:42:57,975 INFO [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "admin" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter] Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy servlet is disabled
2023-12-08 20:42:57,989 INFO [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden property: [key=policyLocation, value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
2023-12-08 20:42:58,455 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b, clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d[ID=f85ec51b<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d[ID=f85ec51b>, local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
2023-12-08 20:42:59,524 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858[ID=7cef23a2<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858[ID=7cef23a2>, local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
2023-12-08 20:42:59,535 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message with ID: 22, result: NO_BINDINGS
2023-12-08 20:42:59,536 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session connectionID: 7cef23a2
2023-12-08 20:42:59,549 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message with ID: 24, result: NO_BINDINGS
2023-12-08 20:42:59,550 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session connectionID: 7cef23a2
2023-12-08 20:42:59,583 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message with ID: 26, result: OK
2023-12-08 20:42:59,584 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created queue: QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=true]@5f46053b
2023-12-08 20:42:59,593 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created consumer with ID: 0, with session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7
2023-12-08 20:42:59,594 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message with ID: 28, result: OK
2023-12-08 20:42:59,616 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent message with ID: 29, result: OK, transaction: UNAVAILABLE
2023-12-08 20:42:59,615 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: acknowledged message: Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0, timestamp=0,expiration=0, durable=false, address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@967407085, with transaction: null
2023-12-08 20:42:59,621 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered message with message ID: 31, to consumer on address: activemq.notifications, queue: notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker, consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
2023-12-08 20:42:59,621 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: acknowledged message: Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0, timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false, address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179613,_AMQ_ClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, with transaction: null
2023-12-08 20:42:59,622 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered message with message ID: 32, to consumer on address: activemq.notifications, queue: notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker, consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
2023-12-08 20:42:59,622 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: acknowledged message: Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0, timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false, address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179614,_AMQ_ClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, with transaction: null
2023-12-08 20:42:59,622 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered message with message ID: 35, to consumer on address: activemq.notifications, queue: notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker, consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
2023-12-08 20:42:59,622 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: acknowledged message: Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0, timestamp=0,expiration=0, durable=false, address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@333704293, with transaction: null
2023-12-08 20:42:59,622 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered message with message ID: 39, to consumer on address: activemq.notifications, queue: notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker, consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
2023-12-08 20:43:08,447 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256[ID=1d1d79ce<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256[ID=1d1d79ce>, local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
2023-12-08 20:43:08,464 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7[ID=8f88c6d5<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7[ID=8f88c6d5>, local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
HA “slave” broker log.
2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server] AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null] started, waiting live to fail before it gets active
2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server] AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is synchronized with live server, nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server] AMQ221031: backup announced
2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server] AMQ221083: ignoring quorum vote as max cluster size is 1.
2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server] AMQ221071: Failing over based on quorum vote results.
2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/10.44.0.20:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/10.44.0.20:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server] AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST]
2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for protocols [MQTT]
2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for protocols [CORE]
2023-12-08 20:42:51,946 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl [ID=df357d0e, clientID=null, nodeID=9759f092-9609-11ee-9862-e26c98ea5364, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e>, local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
2023-12-08 20:42:56,977 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001: destroyed connection: RemotingConnectionImpl [ID=df357d0e, clientID=null, nodeID=9759f092-9609-11ee-9862-e26c98ea5364, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e>, local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server.
2023-12-08 20:42:57,445 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message with ID: 26, result: NO_BINDINGS
2023-12-08 20:42:57,447 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created queue: QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@737e64a6
2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server] AMQ222186: unable to authorise cluster control: AMQ219016: Connection failure detected. Unblocking a blocking call that will never get a response
2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server] AMQ224091: Bridge ClusterConnectionBridge@2a063f52 [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@737e64a6 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52 [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@737e64a6 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008[nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364, connector=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local, address=, server=ActiveMQServerImpl::name=amq-broker])) [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local], discoveryGroupConfiguration=null]] is unable to connect to destination. Retrying
2023-12-08 20:42:57,629 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl [ID=483170f6, clientID=null, nodeID=9759f092-9609-11ee-9862-e26c98ea5364, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251[ID=483170f6<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251[ID=483170f6>, local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
2023-12-08 20:42:57,630 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null, nodeID=9759f092-9609-11ee-9862-e26c98ea5364, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b[ID=4514a2b3<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b[ID=4514a2b3>, local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
2023-12-08 20:42:57,648 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null, nodeID=9759f092-9609-11ee-9862-e26c98ea5364, transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8[ID=fcde4d78<mailto:transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8[ID=fcde4d78>, local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
2023-12-08 20:42:57,663 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message with ID: 29, result: NO_BINDINGS
2023-12-08 20:42:57,663 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session connectionID: fcde4d78
2023-12-08 20:42:57,679 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message with ID: 31, result: NO_BINDINGS
2023-12-08 20:42:57,679 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session connectionID: fcde4d78
2023-12-08 20:42:57,708 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message with ID: 33, result: OK
2023-12-08 20:42:57,709 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created queue: QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=true]@3645037f
2023-12-08 20:42:57,719 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created consumer with ID: 0, with session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
2023-12-08 20:42:57,720 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message with ID: 35, result: OK
2023-12-08 20:42:57,755 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent message with ID: 36, result: OK, transaction: UNAVAILABLE
2023-12-08 20:42:57,753 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: acknowledged message: Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0, timestamp=0,expiration=0, durable=false, address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@2125985563, with transaction: null
2023-12-08 20:42:57,760 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered message with message ID: 38, to consumer on address: activemq.notifications, queue: notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker, consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
2023-12-08 20:42:57,761 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: acknowledged message: Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0, timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false, address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, with transaction: null
2023-12-08 20:42:57,761 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered message with message ID: 41, to consumer on address: activemq.notifications, queue: notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker, consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
2023-12-08 20:42:57,762 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: acknowledged message: Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0, timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false, address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, with transaction: null
2023-12-08 20:42:57,762 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered message with message ID: 43, to consumer on address: activemq.notifications, queue: notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker, consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
2023-12-08 20:42:57,762 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014: acknowledged message: Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0, timestamp=0,expiration=0, durable=false, address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@1828250555, with transaction: null
2023-12-08 20:42:57,762 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered message with message ID: 44, to consumer on address: activemq.notifications, queue: notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker, consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server] AMQ221027: Bridge ClusterConnectionBridge@2a063f52 [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@737e64a6 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52 [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@737e64a6 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008[nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364, connector=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local, address=, server=ActiveMQServerImpl::name=amq-broker])) [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local], discoveryGroupConfiguration=null]] is connected
2023-12-08 20:42:59,625 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message with ID: 46, result: OK
2023-12-08 20:42:59,625 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed message with ID: 48, result: OK
2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server] AMQ222186: unable to authorise cluster control: AMQ219016: Connection failure detected. Unblocking a blocking call that will never get a response
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by Justin Bertram <jb...@apache.org>.
Thanks for the additional information. I believe I see what's happening.
When the primary/master broker is restarted it looks like the journal that
it originally had is now gone which means it creates a new journal and
initializes it with the brand new node ID. Since this node ID doesn't match
the node ID of the backup/slave then fail-back won't occur. Instead the two
brokers just cluster together.
Let's walk through the relevant log entries...
First, when the backup starts:
AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
synchronized with live server, nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
Notice the node ID is 9759f092-9609-11ee-9862-e26c98ea5364. This node ID
would have come from the original primary/master.
However, when the primary is restarted it logs:
AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
[amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
Notice the node ID is 59d00b93-960a-11ee-b571-7ab0027aea5f which doesn't
match the node ID of the backup/slave. Since the node IDs are different
they won't pair together.
I believe your Kubernetes pods need to store their journals on persistent
volumes so that they remain in-tact regardless of what happens to the pod.
When the pod is restarted it can re-attach to the PV and continue using its
journal (which contains the node ID).
Justin
On Wed, Dec 13, 2023 at 6:29 PM Shields, Paul Michael <pa...@hpe.com>
wrote:
> Hi Justin,
>
> Thanks for the response. I will try that example. Yes, I am suspecting
> that it is an environmental issue too. I am running the broker instances
> in Kubernetes pods on two different servers. I have an operator pod that
> creates a statefullSet consisting of the master and slave brokers. The pods
> start up in master first (ss-0) and then the slave is started (ss-1). I
> delete the master pod and the slave goes active. The failover to the slave
> works as expected. My operator pod sees that the master broker pod has
> gone away and then starts it up. I think this is where the issue is. The
> master establishes the cluster connection to the slave (which is now the
> active broker) but it does not seem to interact with the slave instance to
> tell it to no longer be active. The master and slave instances reestablish
> the cluster connection and setup the queues associated with the cluster
> connections. I also see some core messages go to the slave, but I not sure
> what they represent. The master instance just completes its startup and
> goes active. Thus creating two active broker instances, in essence a
> split-brain condition. I have turned on logging as such
> <broker-plugins>
> <broker-plugin
> class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
> <property key="LOG_ALL_EVENTS" value="true"/>
> <property key="LOG_CONNECTION_EVENTS" value="true"/>
> <property key="LOG_SESSION_EVENTS" value="true"/>
> <property key="LOG_CONSUMER_EVENTS" value="true"/>
> <property key="LOG_DELIVERING_EVENTS" value="true"/>
> <property key="LOG_SENDING_EVENTS" value="true"/>
> <property key="LOG_INTERNAL_EVENTS" value="true"/>
> </broker-plugin>
> </broker-plugins>
>
> I have also edited the log4j2.properties file inside the Kubernetes pod
> and set the logger.artemis_server.level=INFO property to TRACE and have
> copious amount of log data. What methods would be involved in the startup
> of the master server that would stage the fail-back? That is what is the
> expected path through the code during startup to have a successful
> fail-back. With that info I may be able to trace what is derailing the
> fail-back. Would the trace output be any use to you to determine what is
> going on?
>
> Would the broker.xml be able to give you enough to attempt to recreate the
> issue? I will attach the master one. The slave only differs by the
> ha-policy config I posted earlier.
>
> I plan to run the example you pointed out on two VM servers without
> Kubernetes being involved to see if I can isolate the issue to a Kubernetes
> pod setup issue?
>
> Thanks Paul
>
>
> kubectl -n dvs exec -it cray-dvs-mqtt-ss-0 -- cat
> /home/jboss/amq-broker/etc/broker.xml
>
> <?xml version='1.0'?>
>
> <!--
>
> Licensed to the Apache Software Foundation (ASF) under one
>
> or more contributor license agreements. See the NOTICE file
>
> distributed with this work for additional information
>
> regarding copyright ownership. The ASF licenses this file
>
> to you under the Apache License, Version 2.0 (the
>
> "License"); you may not use this file except in compliance
>
> with the License. You may obtain a copy of the License at
>
>
>
> http://www.apache.org/licenses/LICENSE-2.0
>
>
>
> Unless required by applicable law or agreed to in writing,
>
> software distributed under the License is distributed on an
>
> "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
>
> KIND, either express or implied. See the License for the
>
> specific language governing permissions and limitations
>
> under the License.
>
> -->
>
>
>
> <configuration xmlns="urn:activemq"
>
> xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
>
> xmlns:xi=http://www.w3.org/2001/XInclude
>
> xsi:schemaLocation="urn:activemq
> /schema/artemis-configuration.xsd">
>
>
>
> <core xmlns="urn:activemq:core" xmlns:xsi=
> http://www.w3.org/2001/XMLSchema-instance
>
> xsi:schemaLocation="urn:activemq:core ">
>
>
>
> <name>amq-broker</name>
>
>
>
> <ha-policy>
>
> <replication>
>
> <master>
>
> <check-for-live-server>true</check-for-live-server>
>
> </master>
>
> </replication>
>
> </ha-policy>
>
>
>
>
>
>
>
> <persistence-enabled>true</persistence-enabled>
>
>
>
> <!-- this could be ASYNCIO, MAPPED, NIO
>
> ASYNCIO: Linux Libaio
>
> MAPPED: mmap files
>
> NIO: Plain Java Files
>
> -->
>
> <journal-type>NIO</journal-type>
>
>
>
> <paging-directory>data/paging</paging-directory>
>
>
>
> <bindings-directory>data/bindings</bindings-directory>
>
>
>
> <journal-directory>data/journal</journal-directory>
>
>
>
>
> <large-messages-directory>data/large-messages</large-messages-directory>
>
>
>
>
>
> <!-- if you want to retain your journal uncomment this following
> configuration.
>
>
>
> This will allow your system to keep 7 days of your data, up to 10G.
> Tweak it accordingly to your use case and capacity.
>
>
>
> it is recommended to use a separate storage unit from the journal
> for performance considerations.
>
>
>
> <journal-retention-directory period="7" unit="DAYS"
> storage-limit="10G">data/retention</journal-retention-directory>
>
>
>
> You can also enable retention by using the argument
> journal-retention on the `artemis create` command -->
>
>
>
>
>
>
>
> <journal-datasync>true</journal-datasync>
>
>
>
> <journal-min-files>2</journal-min-files>
>
>
>
> <journal-pool-files>10</journal-pool-files>
>
>
>
> <journal-device-block-size>4096</journal-device-block-size>
>
>
>
> <journal-file-size>10M</journal-file-size>
>
> <!--
>
> You can verify the network health of a particular NIC by
> specifying the <network-check-NIC> element.
>
> <network-check-NIC>theNicName</network-check-NIC>
>
> -->
>
>
>
> <!--
>
> Use this to use an HTTP server to validate the network
>
> <network-check-URL-list>http://www.apache.org
> </network-check-URL-list<http://www.apache.org%3c/network-check-URL-list>>
> -->
>
>
>
> <!-- <network-check-period>10000</network-check-period> -->
>
> <!-- <network-check-timeout>1000</network-check-timeout> -->
>
>
>
> <!-- this is a comma separated list, no spaces, just DNS or IPs
>
> it should accept IPV6
>
>
>
> Warning: Make sure you understand your network topology as this
> is meant to validate if your network is valid.
>
> Using IPs that could eventually disappear or be
> partially visible may defeat the purpose.
>
> You can use a list of multiple IPs, and if any
> successful ping will make the server OK to continue running -->
>
> <!-- <network-check-list>10.0.0.1</network-check-list> -->
>
>
>
> <!-- use this to customize the ping used for ipv4 addresses -->
>
> <!-- <network-check-ping-command>ping -c 1 -t %d
> %s</network-check-ping-command> -->
>
>
>
> <!-- use this to customize the ping used for ipv6 addresses -->
>
> <!-- <network-check-ping6-command>ping6 -c 1
> %2$s</network-check-ping6-command> -->
>
>
>
>
>
>
>
> <connectors>
>
> <!-- Connector used to be announced through cluster connections
> and notifications -->
>
> <connector
> name="artemis">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616</connector>
>
> </connectors>
>
>
>
>
>
>
>
> <!-- how often we are looking for how many bytes are being used on
> the disk in ms -->
>
> <disk-scan-period>5000</disk-scan-period>
>
>
>
> <!-- once the disk hits this limit the system will block, or close
> the connection in certain protocols
>
> that won't support flow control. -->
>
> <max-disk-usage>90</max-disk-usage>
>
>
>
> <!-- should the broker detect dead locks and other issues -->
>
> <critical-analyzer>true</critical-analyzer>
>
>
>
> <critical-analyzer-timeout>120000</critical-analyzer-timeout>
>
>
>
>
> <critical-analyzer-check-period>60000</critical-analyzer-check-period>
>
>
>
> <critical-analyzer-policy>HALT</critical-analyzer-policy>
>
>
>
>
>
>
>
> <!-- the system will enter into page mode once you hit this limit.
> This is an estimate in bytes of how much the messages are using in memory
>
>
>
> The system will use half of the available memory (-Xmx) by default
> for the global-max-size.
>
> You may specify a different value here if you need to customize it
> to your needs.
>
>
>
> <global-max-size>100Mb</global-max-size> -->
>
>
>
> <!-- the maximum number of messages accepted before entering full
> address mode.
>
> if global-max-size is specified the full address mode will be
> specified by whatever hits it first. -->
>
> <global-max-messages>-1</global-max-messages>
>
>
>
> <acceptors>
>
>
>
> <!-- useEpoll means: it will use Netty epoll if you are on a
> system (Linux) that supports it -->
>
> <!-- amqpCredits: The number of credits sent to AMQP producers -->
>
> <!-- amqpLowCredits: The server will send the # credits specified
> at amqpCredits at this low mark -->
>
> <!-- amqpDuplicateDetection: If you are not using duplicate
> detection, set this to false
>
> as duplicate detection requires
> applicationProperties to be parsed on the server. -->
>
> <!-- amqpMinLargeMessageSize: Determines how many bytes are
> considered large, so we start using files to hold their data.
>
> default: 102400, -1 would mean to
> disable large mesasge control -->
>
>
>
> <!-- Note: If an acceptor needs to be compatible with HornetQ
> and/or Artemis 1.x clients add
>
> "anycastPrefix=jms.queue.;multicastPrefix=jms.topic."
> to the acceptor url.
>
> See https://issues.apache.org/jira/browse/ARTEMIS-1644
> for more information. -->
>
>
>
>
>
> <!-- Acceptor for every supported protocol -->
>
>
>
> <acceptor
> name="dvs">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883?protocols=MQTT;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor><acceptor
> name="scaleDown">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616?protocols=CORE;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor>
>
> </acceptors>
>
>
>
>
>
> <cluster-user>EtBU3LYU</cluster-user>
>
>
>
> <cluster-password>xOKqDzwH</cluster-password>
>
>
>
> <broadcast-groups>
>
> <broadcast-group name="my-broadcast-group">
> <jgroups-file>jgroups-ping.xml</jgroups-file>
> <jgroups-channel>activemq_broadcast_channel</jgroups-channel>
> <connector-ref>artemis</connector-ref> </broadcast-group>
>
> </broadcast-groups>
>
>
>
> <discovery-groups>
>
> <discovery-group name="my-discovery-group">
> <jgroups-file>jgroups-ping.xml</jgroups-file>
> <jgroups-channel>activemq_broadcast_channel</jgroups-channel>
> <refresh-timeout>10000</refresh-timeout> </discovery-group>
>
> </discovery-groups>
>
>
>
> <cluster-connections>
>
> <cluster-connection name="my-cluster">
> <connector-ref>artemis</connector-ref>
> <retry-interval>1000</retry-interval>
> <retry-interval-multiplier>2</retry-interval-multiplier>
> <max-retry-interval>32000</max-retry-interval>
> <initial-connect-attempts>20</initial-connect-attempts>
> <reconnect-attempts>-1</reconnect-attempts>
> <use-duplicate-detection>true</use-duplicate-detection>
> <message-load-balancing>ON_DEMAND</message-load-balancing>
> <max-hops>1</max-hops> <discovery-group-ref
> discovery-group-name="my-discovery-group"/> </cluster-connection>
>
> </cluster-connections>
>
>
>
>
>
> <security-settings>
>
> <security-setting match="#">
>
> <permission type="createNonDurableQueue" roles="admin"/>
>
> <permission type="deleteNonDurableQueue" roles="admin"/>
>
> <permission type="createDurableQueue" roles="admin"/>
>
> <permission type="deleteDurableQueue" roles="admin"/>
>
> <permission type="createAddress" roles="admin"/>
>
> <permission type="deleteAddress" roles="admin"/>
>
> <permission type="consume" roles="admin"/>
>
> <permission type="browse" roles="admin"/>
>
> <permission type="send" roles="admin"/>
>
> <!-- we need this otherwise ./artemis data imp wouldn't work
> -->
>
> <permission type="manage" roles="admin"/>
>
> </security-setting>
>
> </security-settings>
>
>
>
> <address-settings>
>
> <!-- if you define auto-create on certain queues, management has
> to be auto-create -->
>
> <address-setting match="activemq.management#">
>
> <dead-letter-address>DLQ</dead-letter-address>
>
> <expiry-address>ExpiryQueue</expiry-address>
>
> <redelivery-delay>0</redelivery-delay>
>
> <!-- with -1 only the global-max-size is in use for limiting
> -->
>
> <max-size-bytes>-1</max-size-bytes>
>
>
> <message-counter-history-day-limit>10</message-counter-history-day-limit>
>
> <address-full-policy>PAGE</address-full-policy>
>
> <auto-create-queues>true</auto-create-queues>
>
> <auto-create-addresses>true</auto-create-addresses>
>
> </address-setting>
>
> <!--default for catch all-->
>
> <address-setting match="#">
>
> <redistribution-delay>0</redistribution-delay>
>
> <dead-letter-address>DLQ</dead-letter-address>
>
> <expiry-address>ExpiryQueue</expiry-address>
>
> <redelivery-delay>0</redelivery-delay>
>
>
>
> <!-- if max-size-bytes and max-size-messages were both
> enabled, the system will enter into paging
>
> based on the first attribute to hits the maximum value -->
>
> <!-- limit for the address in bytes, -1 means unlimited -->
>
> <max-size-bytes>-1</max-size-bytes>
>
>
>
> <!-- limit for the address in messages, -1 means unlimited -->
>
> <max-size-messages>-1</max-size-messages>
>
>
>
> <!-- the size of each file on paging. Notice we keep files in
> memory while they are in use.
>
> Lower this setting if you have too many queues in memory.
> -->
>
> <page-size-bytes>10M</page-size-bytes>
>
>
>
> <!-- limit how many messages are read from paging into the
> Queue. -->
>
> <max-read-page-messages>-1</max-read-page-messages>
>
>
>
> <!-- limit how much memory is read from paging into the Queue.
> -->
>
> <max-read-page-bytes>20M</max-read-page-bytes>
>
>
>
>
> <message-counter-history-day-limit>10</message-counter-history-day-limit>
>
> <address-full-policy>PAGE</address-full-policy>
>
> <auto-create-queues>true</auto-create-queues>
>
> <auto-create-addresses>true</auto-create-addresses>
>
> <auto-delete-queues>false</auto-delete-queues>
>
> <auto-delete-addresses>false</auto-delete-addresses>
>
> </address-setting>
>
> </address-settings>
>
>
>
> <addresses>
>
> <address name="DLQ">
>
> <anycast>
>
> <queue name="DLQ" />
>
> </anycast>
>
> </address>
>
> <address name="ExpiryQueue">
>
> <anycast>
>
> <queue name="ExpiryQueue" />
>
> </anycast>
>
> </address>
>
>
>
> </addresses>
>
>
>
>
>
>
>
> <broker-plugins>
>
> <broker-plugin
> class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
>
> <property key="LOG_ALL_EVENTS" value="true"/>
>
> <property key="LOG_CONNECTION_EVENTS" value="true"/>
>
> <property key="LOG_SESSION_EVENTS" value="true"/>
>
> <property key="LOG_CONSUMER_EVENTS" value="true"/>
>
> <property key="LOG_DELIVERING_EVENTS" value="true"/>
>
> <property key="LOG_SENDING_EVENTS" value="true"/>
>
> <property key="LOG_INTERNAL_EVENTS" value="true"/>
>
> </broker-plugin>
>
> </broker-plugins>
>
>
>
> <metrics> <plugin
> class-name="com.redhat.amq.broker.core.server.metrics.plugins.ArtemisPrometheusMetricsPlugin"/>
> </metrics>
>
> </core>
>
> </configuration>
>
> From: Justin Bertram <jb...@apache.org>
> Date: Wednesday, December 13, 2023 at 3:23 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> ActiveMQ Artemis ships with an example in the
> examples/features/ha/replicated-failback directory that demonstrates how
> this works. I ran this example and it worked fine. I also started up the
> brokers manually and triggered failover and failback. Again, everything
> worked fine. My only guess is that there's something configured wrong or
> perhaps there's an environmental issue that's preventing failback from
> working properly for you. If you could provide clear steps on how to
> reproduce this issue I could investigate further.
>
>
> Justin
>
> On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <paul.shields@hpe.com
> >
> wrote:
>
> > Hi,
> >
> > I am having an issue with failback to the master broker. I start the
> > master broker then the slave broker. I see the replication from the
> master
> > to the slave happen. I stop the master/active broker and a failover
> happens
> > to the slave as expected. When the master is started a failback to it is
> > expected but does not happen. The following is my ha-policy config.
> Did I
> > miss something in the ha-policy configuration? We are using version
> 2.28.0
> > and java17.
> >
> > Thanks,
> > Paul
> >
> >
> > Master broker.xml config
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <master>
> >
> > <check-for-live-server>true</check-for-live-server>
> >
> > </master>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > Slave broker.xml config
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > The following are the logs from the master and slave brokers
> > HA “master” broker log after restart
> > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> > protocols [MQTT]
> > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> > protocols [CORE]
> > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221007: Server is now live
> > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > Starting embedded web server
> > 2023-12-08 20:42:57,625 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 15, result: NO_BINDINGS
> > 2023-12-08 20:42:57,627 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> > queue:
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@15525ed3
> > 2023-12-08 20:42:57,657 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > [ID=47edf16f<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> [ID=47edf16f>,
> > local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
> > 2023-12-08 20:42:57,720 INFO
> > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > activemq-branding plugin
> > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > connector=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]] is connected
> > 2023-12-08 20:42:57,755 INFO
> > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > artemis-plugin plugin
> > 2023-12-08 20:42:57,766 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 18, result: NO_BINDINGS
> > 2023-12-08 20:42:57,767 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 20, result: NO_BINDINGS
> > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> Initialising
> > hawtio services
> > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> Configuration
> > will be discovered via system properties
> > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > Hawtio 2.15.0
> > 2023-12-08 20:42:57,975 INFO
> > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > authentication filter, JAAS realm: "activemq" authorized role(s): "admin"
> > role principal classes:
> > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > servlet is disabled
> > 2023-12-08 20:42:57,989 INFO
> > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > property: [key=policyLocation,
> > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > HTTP Server started at
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > Artemis Jolokia REST API available at
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > 2023-12-08
> > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> >
> > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> Console
> > available at
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > 2023-12-08
> > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> > 20:42:58,455 INFO [org.apache.activemq.artemis.core.server.plugin.impl]
> > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > [ID=f85ec51b<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> [ID=f85ec51b>,
> > local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
> > 2023-12-08 20:42:59,524 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > [ID=7cef23a2<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> [ID=7cef23a2>,
> > local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
> > 2023-12-08 20:42:59,535 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 22, result: NO_BINDINGS
> > 2023-12-08 20:42:59,536 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session connectionID:
> > 7cef23a2
> > 2023-12-08 20:42:59,549 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 24, result: NO_BINDINGS
> > 2023-12-08 20:42:59,550 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session connectionID:
> > 7cef23a2
> > 2023-12-08 20:42:59,583 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 26, result: OK
> > 2023-12-08 20:42:59,584 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> > queue:
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=true]@5f46053b
> > 2023-12-08 20:42:59,593 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> > consumer with ID: 0, with session name:
> 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > 2023-12-08 20:42:59,594 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 28, result: OK
> > 2023-12-08 20:42:59,616 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > 2023-12-08 20:42:59,615 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > timestamp=0,expiration=0, durable=false,
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@967407085,
> > with transaction: null
> > 2023-12-08 20:42:59,621 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 31, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > 2023-12-08 20:42:59,621 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179613,_AMQ_ClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
> > with transaction: null
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 32, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179614,_AMQ_ClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
> > with transaction: null
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 35, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > timestamp=0,expiration=0, durable=false,
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@333704293,
> > with transaction: null
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 39, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > 2023-12-08 20:43:08,447 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > [ID=1d1d79ce<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> [ID=1d1d79ce>,
> > local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
> > 2023-12-08 20:43:08,464 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > [ID=8f88c6d5<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> [ID=8f88c6d5>,
> > local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
> >
> >
> > HA “slave” broker log.
> >
> > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > synchronized with live server,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> >
> > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221031: backup announced
> > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221084: Requested 0 quorum votes
> > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221071: Failing over based on quorum vote results.
> > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > AMQ212037: Connection failure to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> > disconnected because of server shutdown [code=DISCONNECTED]
> > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > AMQ212037: Connection failure to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> > disconnected because of server shutdown [code=DISCONNECTED]
> > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221007: Server is now live
> > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at
> > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> > protocols [MQTT]
> > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at
> > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> > protocols [CORE]
> > 2023-12-08 20:42:51,946 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > 2023-12-08 20:42:56,977 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> destroyed
> > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > AMQ212004: Failed to connect to server.
> > 2023-12-08 20:42:57,445 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 26, result: NO_BINDINGS
> > 2023-12-08 20:42:57,447 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> > queue:
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6
> > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > failure detected. Unblocking a blocking call that will never get a
> response
> > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > connector=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > Retrying
> > 2023-12-08 20:42:57,629 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> [ID=483170f6>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
> > 2023-12-08 20:42:57,630 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> [ID=4514a2b3>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
> > 2023-12-08 20:42:57,648 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> [ID=fcde4d78>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
> > 2023-12-08 20:42:57,663 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 29, result: NO_BINDINGS
> > 2023-12-08 20:42:57,663 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session connectionID:
> > fcde4d78
> > 2023-12-08 20:42:57,679 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 31, result: NO_BINDINGS
> > 2023-12-08 20:42:57,679 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session connectionID:
> > fcde4d78
> > 2023-12-08 20:42:57,708 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 33, result: OK
> > 2023-12-08 20:42:57,709 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> > queue:
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=true]@3645037f
> > 2023-12-08 20:42:57,719 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> > consumer with ID: 0, with session name:
> 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > 2023-12-08 20:42:57,720 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 35, result: OK
> > 2023-12-08 20:42:57,755 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > 2023-12-08 20:42:57,753 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > timestamp=0,expiration=0, durable=false,
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@2125985563,
> > with transaction: null
> > 2023-12-08 20:42:57,760 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 38, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > 2023-12-08 20:42:57,761 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
> > with transaction: null
> > 2023-12-08 20:42:57,761 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 41, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > 2023-12-08 20:42:57,762 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
> > with transaction: null
> > 2023-12-08 20:42:57,762 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 43, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > 2023-12-08 20:42:57,762 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > timestamp=0,expiration=0, durable=false,
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@1828250555,
> > with transaction: null
> > 2023-12-08 20:42:57,762 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 44, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > connector=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]] is connected
> > 2023-12-08 20:42:59,625 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 46, result: OK
> > 2023-12-08 20:42:59,625 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 48, result: OK
> > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > failure detected. Unblocking a blocking call that will never get a
> response
> >
> >
> >
>
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by "Shields, Paul Michael" <pa...@hpe.com>.
Hi Justin,
Thanks for the response. I will try that example. Yes, I am suspecting that it is an environmental issue too. I am running the broker instances in Kubernetes pods on two different servers. I have an operator pod that creates a statefullSet consisting of the master and slave brokers. The pods start up in master first (ss-0) and then the slave is started (ss-1). I delete the master pod and the slave goes active. The failover to the slave works as expected. My operator pod sees that the master broker pod has gone away and then starts it up. I think this is where the issue is. The master establishes the cluster connection to the slave (which is now the active broker) but it does not seem to interact with the slave instance to tell it to no longer be active. The master and slave instances reestablish the cluster connection and setup the queues associated with the cluster connections. I also see some core messages go to the slave, but I not sure what they represent. The master instance just completes its startup and goes active. Thus creating two active broker instances, in essence a split-brain condition. I have turned on logging as such
<broker-plugins>
<broker-plugin class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
<property key="LOG_ALL_EVENTS" value="true"/>
<property key="LOG_CONNECTION_EVENTS" value="true"/>
<property key="LOG_SESSION_EVENTS" value="true"/>
<property key="LOG_CONSUMER_EVENTS" value="true"/>
<property key="LOG_DELIVERING_EVENTS" value="true"/>
<property key="LOG_SENDING_EVENTS" value="true"/>
<property key="LOG_INTERNAL_EVENTS" value="true"/>
</broker-plugin>
</broker-plugins>
I have also edited the log4j2.properties file inside the Kubernetes pod and set the logger.artemis_server.level=INFO property to TRACE and have copious amount of log data. What methods would be involved in the startup of the master server that would stage the fail-back? That is what is the expected path through the code during startup to have a successful fail-back. With that info I may be able to trace what is derailing the fail-back. Would the trace output be any use to you to determine what is going on?
Would the broker.xml be able to give you enough to attempt to recreate the issue? I will attach the master one. The slave only differs by the ha-policy config I posted earlier.
I plan to run the example you pointed out on two VM servers without Kubernetes being involved to see if I can isolate the issue to a Kubernetes pod setup issue?
Thanks Paul
kubectl -n dvs exec -it cray-dvs-mqtt-ss-0 -- cat /home/jboss/amq-broker/etc/broker.xml
<?xml version='1.0'?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<configuration xmlns="urn:activemq"
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
xmlns:xi=http://www.w3.org/2001/XInclude
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
xsi:schemaLocation="urn:activemq:core ">
<name>amq-broker</name>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<persistence-enabled>true</persistence-enabled>
<!-- this could be ASYNCIO, MAPPED, NIO
ASYNCIO: Linux Libaio
MAPPED: mmap files
NIO: Plain Java Files
-->
<journal-type>NIO</journal-type>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<!-- if you want to retain your journal uncomment this following configuration.
This will allow your system to keep 7 days of your data, up to 10G. Tweak it accordingly to your use case and capacity.
it is recommended to use a separate storage unit from the journal for performance considerations.
<journal-retention-directory period="7" unit="DAYS" storage-limit="10G">data/retention</journal-retention-directory>
You can also enable retention by using the argument journal-retention on the `artemis create` command -->
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<!--
You can verify the network health of a particular NIC by specifying the <network-check-NIC> element.
<network-check-NIC>theNicName</network-check-NIC>
-->
<!--
Use this to use an HTTP server to validate the network
<network-check-URL-list>http://www.apache.org</network-check-URL-list<http://www.apache.org%3c/network-check-URL-list>> -->
<!-- <network-check-period>10000</network-check-period> -->
<!-- <network-check-timeout>1000</network-check-timeout> -->
<!-- this is a comma separated list, no spaces, just DNS or IPs
it should accept IPV6
Warning: Make sure you understand your network topology as this is meant to validate if your network is valid.
Using IPs that could eventually disappear or be partially visible may defeat the purpose.
You can use a list of multiple IPs, and if any successful ping will make the server OK to continue running -->
<!-- <network-check-list>10.0.0.1</network-check-list> -->
<!-- use this to customize the ping used for ipv4 addresses -->
<!-- <network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command> -->
<!-- use this to customize the ping used for ipv6 addresses -->
<!-- <network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command> -->
<connectors>
<!-- Connector used to be announced through cluster connections and notifications -->
<connector name="artemis">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616</connector>
</connectors>
<!-- how often we are looking for how many bytes are being used on the disk in ms -->
<disk-scan-period>5000</disk-scan-period>
<!-- once the disk hits this limit the system will block, or close the connection in certain protocols
that won't support flow control. -->
<max-disk-usage>90</max-disk-usage>
<!-- should the broker detect dead locks and other issues -->
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<!-- the system will enter into page mode once you hit this limit. This is an estimate in bytes of how much the messages are using in memory
The system will use half of the available memory (-Xmx) by default for the global-max-size.
You may specify a different value here if you need to customize it to your needs.
<global-max-size>100Mb</global-max-size> -->
<!-- the maximum number of messages accepted before entering full address mode.
if global-max-size is specified the full address mode will be specified by whatever hits it first. -->
<global-max-messages>-1</global-max-messages>
<acceptors>
<!-- useEpoll means: it will use Netty epoll if you are on a system (Linux) that supports it -->
<!-- amqpCredits: The number of credits sent to AMQP producers -->
<!-- amqpLowCredits: The server will send the # credits specified at amqpCredits at this low mark -->
<!-- amqpDuplicateDetection: If you are not using duplicate detection, set this to false
as duplicate detection requires applicationProperties to be parsed on the server. -->
<!-- amqpMinLargeMessageSize: Determines how many bytes are considered large, so we start using files to hold their data.
default: 102400, -1 would mean to disable large mesasge control -->
<!-- Note: If an acceptor needs to be compatible with HornetQ and/or Artemis 1.x clients add
"anycastPrefix=jms.queue.;multicastPrefix=jms.topic." to the acceptor url.
See https://issues.apache.org/jira/browse/ARTEMIS-1644 for more information. -->
<!-- Acceptor for every supported protocol -->
<acceptor name="dvs">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883?protocols=MQTT;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor><acceptor name="scaleDown">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616?protocols=CORE;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor>
</acceptors>
<cluster-user>EtBU3LYU</cluster-user>
<cluster-password>xOKqDzwH</cluster-password>
<broadcast-groups>
<broadcast-group name="my-broadcast-group"> <jgroups-file>jgroups-ping.xml</jgroups-file> <jgroups-channel>activemq_broadcast_channel</jgroups-channel> <connector-ref>artemis</connector-ref> </broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group"> <jgroups-file>jgroups-ping.xml</jgroups-file> <jgroups-channel>activemq_broadcast_channel</jgroups-channel> <refresh-timeout>10000</refresh-timeout> </discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster"> <connector-ref>artemis</connector-ref> <retry-interval>1000</retry-interval> <retry-interval-multiplier>2</retry-interval-multiplier> <max-retry-interval>32000</max-retry-interval> <initial-connect-attempts>20</initial-connect-attempts> <reconnect-attempts>-1</reconnect-attempts> <use-duplicate-detection>true</use-duplicate-detection> <message-load-balancing>ON_DEMAND</message-load-balancing> <max-hops>1</max-hops> <discovery-group-ref discovery-group-name="my-discovery-group"/> </cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="admin"/>
<permission type="deleteNonDurableQueue" roles="admin"/>
<permission type="createDurableQueue" roles="admin"/>
<permission type="deleteDurableQueue" roles="admin"/>
<permission type="createAddress" roles="admin"/>
<permission type="deleteAddress" roles="admin"/>
<permission type="consume" roles="admin"/>
<permission type="browse" roles="admin"/>
<permission type="send" roles="admin"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="admin"/>
</security-setting>
</security-settings>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<redistribution-delay>0</redistribution-delay>
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- if max-size-bytes and max-size-messages were both enabled, the system will enter into paging
based on the first attribute to hits the maximum value -->
<!-- limit for the address in bytes, -1 means unlimited -->
<max-size-bytes>-1</max-size-bytes>
<!-- limit for the address in messages, -1 means unlimited -->
<max-size-messages>-1</max-size-messages>
<!-- the size of each file on paging. Notice we keep files in memory while they are in use.
Lower this setting if you have too many queues in memory. -->
<page-size-bytes>10M</page-size-bytes>
<!-- limit how many messages are read from paging into the Queue. -->
<max-read-page-messages>-1</max-read-page-messages>
<!-- limit how much memory is read from paging into the Queue. -->
<max-read-page-bytes>20M</max-read-page-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-delete-queues>false</auto-delete-queues>
<auto-delete-addresses>false</auto-delete-addresses>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
<broker-plugins>
<broker-plugin class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
<property key="LOG_ALL_EVENTS" value="true"/>
<property key="LOG_CONNECTION_EVENTS" value="true"/>
<property key="LOG_SESSION_EVENTS" value="true"/>
<property key="LOG_CONSUMER_EVENTS" value="true"/>
<property key="LOG_DELIVERING_EVENTS" value="true"/>
<property key="LOG_SENDING_EVENTS" value="true"/>
<property key="LOG_INTERNAL_EVENTS" value="true"/>
</broker-plugin>
</broker-plugins>
<metrics> <plugin class-name="com.redhat.amq.broker.core.server.metrics.plugins.ArtemisPrometheusMetricsPlugin"/> </metrics>
</core>
</configuration>
From: Justin Bertram <jb...@apache.org>
Date: Wednesday, December 13, 2023 at 3:23 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
ActiveMQ Artemis ships with an example in the
examples/features/ha/replicated-failback directory that demonstrates how
this works. I ran this example and it worked fine. I also started up the
brokers manually and triggered failover and failback. Again, everything
worked fine. My only guess is that there's something configured wrong or
perhaps there's an environmental issue that's preventing failback from
working properly for you. If you could provide clear steps on how to
reproduce this issue I could investigate further.
Justin
On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <pa...@hpe.com>
wrote:
> Hi,
>
> I am having an issue with failback to the master broker. I start the
> master broker then the slave broker. I see the replication from the master
> to the slave happen. I stop the master/active broker and a failover happens
> to the slave as expected. When the master is started a failback to it is
> expected but does not happen. The following is my ha-policy config. Did I
> miss something in the ha-policy configuration? We are using version 2.28.0
> and java17.
>
> Thanks,
> Paul
>
>
> Master broker.xml config
>
> <ha-policy>
>
> <replication>
>
> <master>
>
> <check-for-live-server>true</check-for-live-server>
>
> </master>
>
> </replication>
>
> </ha-policy>
>
> Slave broker.xml config
>
> <ha-policy>
>
> <replication>
>
> <slave>
>
> <allow-failback>true</allow-failback>
>
> </slave>
>
> </replication>
>
> </ha-policy>
>
> The following are the logs from the master and slave brokers
> HA “master” broker log after restart
> 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> protocols [MQTT]
> 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> protocols [CORE]
> 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> AMQ221007: Server is now live
> 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> Starting embedded web server
> 2023-12-08 20:42:57,625 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 15, result: NO_BINDINGS
> 2023-12-08 20:42:57,627 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@15525ed3
> 2023-12-08 20:42:57,657 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> [ID=47edf16f<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2[ID=47edf16f>,
> local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
> 2023-12-08 20:42:57,720 INFO
> [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> activemq-branding plugin
> 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883[nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> connector=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> address=, server=ActiveMQServerImpl::name=amq-broker]))
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]] is connected
> 2023-12-08 20:42:57,755 INFO
> [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> artemis-plugin plugin
> 2023-12-08 20:42:57,766 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 18, result: NO_BINDINGS
> 2023-12-08 20:42:57,767 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 20, result: NO_BINDINGS
> 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener] Initialising
> hawtio services
> 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager] Configuration
> will be discovered via system properties
> 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> Hawtio 2.15.0
> 2023-12-08 20:42:57,975 INFO
> [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> authentication filter, JAAS realm: "activemq" authorized role(s): "admin"
> role principal classes:
> "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> servlet is disabled
> 2023-12-08 20:42:57,989 INFO
> [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> property: [key=policyLocation,
> value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> HTTP Server started at
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> Artemis Jolokia REST API available at
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> 2023-12-08
> <http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 >
> 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console
> available at
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> 2023-12-08
> <http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08 >
> 20:42:58,455 INFO [org.apache.activemq.artemis.core.server.plugin.impl]
> AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> [ID=f85ec51b<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d[ID=f85ec51b>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
> 2023-12-08 20:42:59,524 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> [ID=7cef23a2<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858[ID=7cef23a2>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
> 2023-12-08 20:42:59,535 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 22, result: NO_BINDINGS
> 2023-12-08 20:42:59,536 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session connectionID:
> 7cef23a2
> 2023-12-08 20:42:59,549 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 24, result: NO_BINDINGS
> 2023-12-08 20:42:59,550 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session connectionID:
> 7cef23a2
> 2023-12-08 20:42:59,583 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 26, result: OK
> 2023-12-08 20:42:59,584 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=true]@5f46053b
> 2023-12-08 20:42:59,593 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> consumer with ID: 0, with session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7
> 2023-12-08 20:42:59,594 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 28, result: OK
> 2023-12-08 20:42:59,616 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> message with ID: 29, result: OK, transaction: UNAVAILABLE
> 2023-12-08 20:42:59,615 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@967407085,
> with transaction: null
> 2023-12-08 20:42:59,621 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 31, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:42:59,621 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179613,_AMQ_ClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
> with transaction: null
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 32, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179614,_AMQ_ClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
> with transaction: null
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 35, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@333704293,
> with transaction: null
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 39, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:43:08,447 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> [ID=1d1d79ce<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256[ID=1d1d79ce>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
> 2023-12-08 20:43:08,464 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> [ID=8f88c6d5<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7[ID=8f88c6d5>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
>
>
> HA “slave” broker log.
>
> 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> synchronized with live server, nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
>
> 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> AMQ221031: backup announced
> 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> AMQ221084: Requested 0 quorum votes
> 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> AMQ221083: ignoring quorum vote as max cluster size is 1.
> 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> AMQ221071: Failing over based on quorum vote results.
> 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> AMQ212037: Connection failure to
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> disconnected because of server shutdown [code=DISCONNECTED]
> 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> AMQ212037: Connection failure to
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> disconnected because of server shutdown [code=DISCONNECTED]
> 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address DLQ supporting [ANYCAST]
> 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> AMQ221007: Server is now live
> 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> protocols [MQTT]
> 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> protocols [CORE]
> 2023-12-08 20:42:51,946 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> 2023-12-08 20:42:56,977 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001: destroyed
> connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> AMQ212004: Failed to connect to server.
> 2023-12-08 20:42:57,445 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 26, result: NO_BINDINGS
> 2023-12-08 20:42:57,447 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6
> 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> failure detected. Unblocking a blocking call that will never get a response
> 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008[nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> connector=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> address=, server=ActiveMQServerImpl::name=amq-broker]))
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]] is unable to connect to destination.
> Retrying
> 2023-12-08 20:42:57,629 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> [ID=483170f6<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251[ID=483170f6>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
> 2023-12-08 20:42:57,630 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> [ID=4514a2b3<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b[ID=4514a2b3>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
> 2023-12-08 20:42:57,648 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> [ID=fcde4d78<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8[ID=fcde4d78>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
> 2023-12-08 20:42:57,663 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 29, result: NO_BINDINGS
> 2023-12-08 20:42:57,663 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session connectionID:
> fcde4d78
> 2023-12-08 20:42:57,679 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 31, result: NO_BINDINGS
> 2023-12-08 20:42:57,679 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session connectionID:
> fcde4d78
> 2023-12-08 20:42:57,708 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 33, result: OK
> 2023-12-08 20:42:57,709 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=true]@3645037f
> 2023-12-08 20:42:57,719 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> consumer with ID: 0, with session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> 2023-12-08 20:42:57,720 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 35, result: OK
> 2023-12-08 20:42:57,755 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> message with ID: 36, result: OK, transaction: UNAVAILABLE
> 2023-12-08 20:42:57,753 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@2125985563,
> with transaction: null
> 2023-12-08 20:42:57,760 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 38, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:57,761 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
> with transaction: null
> 2023-12-08 20:42:57,761 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 41, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
> with transaction: null
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 43, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@1828250555,
> with transaction: null
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 44, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008[nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> connector=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> address=, server=ActiveMQServerImpl::name=amq-broker]))
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]] is connected
> 2023-12-08 20:42:59,625 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 46, result: OK
> 2023-12-08 20:42:59,625 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 48, result: OK
> 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> failure detected. Unblocking a blocking call that will never get a response
>
>
>
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by "Shields, Paul Michael" <pa...@hpe.com>.
Thanks Justin! I will make that change and give it a try.
Paul
From: Justin Bertram <jb...@apache.org>
Date: Monday, January 15, 2024 at 10:56 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
In the replication case you still need a PV in K8s, but I don't believe you need to put the whole journal on the PV (and suffer the performance penalty). You should just need to point the node-manager-lock-directory in broker.xml to the PV.
Justin
On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <li...@redpointglobal.com.invalid> wrote:
Hi Justin,
Based on your reply in the provided link, are you saying that even for the replication HA case, you still need an external FileShare PV, in K8s, so that the nodeID can be persisted between restarts. In this use case, is the performance of this FileShare less of a concern than it is for the shared-store use case?
Thanks,
Lino
[rg]<https://www.redpointglobal.com/>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
-----Original Message-----
From: Justin Bertram <jb...@apache.org>>
Sent: Friday, January 12, 2024 4:53 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
I sent that email on December 15. You can find it on the web-based mailing
list interface [1].
Justin
[1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l<https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l>
On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <pa...@hpe.com>>
wrote:
> Hi Justin,
>
> Forgive me but I don’t remember a journal persistence issue discussion. I
> did an email search too and did not see that discussion either. Would you
> please refresh me on the discussion? I may have inadvertently deleted that?
>
> I do see data being replicated to my production slave/standby instance on
> initial startup of the two broker instances. Failover works as expected.
> But when the primary instance starts backup I do not see the active slave
> replicate to the starting primary and I do not see the active slave restart
> and go to the backup server state.
>
> What is the communication sequence when the primary starts up after it
> quit with the active backup broker? In other words, how is the active
> backup broker notified to start replication to the newly started primary
> broker?
>
> I ran the replicated-failback example again with TRACE level logging
> turned on and I do see the attemptFailback=false message in the
> target/server1/log/artemis.log file at startup, so it must be changed later
> or am I not understanding the purpose of this flag?
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> discoveryInitialWaitTimeout=10000}]
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost, b=null],
> nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost)
>
> From my production slave broker log
>
>
> 024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> Thanks,
> Paul
> From: Justin Bertram <jb...@apache.org>>
> Date: Friday, January 12, 2024 at 3:38 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > Is it possible to use jgroups instead of UDP for the cluster connection
> in the example?
>
> Yes. Check out examples/features/clustered/clustered-jgroups for an example
> configuration. Keep in mind that UDP, JGroups, etc. is only used for
> initial discovery. Once brokers are discovered then normal TCP connections
> are created.
>
> Before I continue looking through your logs, etc. can you confirm that you
> resolved the journal persistence issue I described in my previous email? If
> that's not resolved there's no chance for failback to occur.
>
>
> Justin
>
>
>
> On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>>
> wrote:
>
> > Hi Justin,
> >
> > I tried the example and it works. I turned up logging to get a success
> > path through the code to maybe see where things get de-railed. Is it
> > possible to use jgroups instead of UDP for the cluster connection in the
> > example? That would give me a better success signature.
> >
> > My standby instance has this in the broker.xml which is the same as in
> the
> > example:
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > I am seeing this series of log messages that suggests that failback is
> not
> > enabled. Is the attemptFailback=false the default at this point in
> startup
> > and it is changed later or it there something gone wrong at this point?
> >
> > 2024-01-11 18:24:14,066 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > QuorumManager(server=null) about node up =
> > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> > <mailto:
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> >
> > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > ClientSessionFactoryImpl received backup update for live/backup pair =
> > TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > / null but it didn't belong to TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,067 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Cluster Connected
> >
> > 2024-01-11 18:24:14,068 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting backup manager
> >
> > 2024-01-11 18:24:14,069 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > backup config ClusterConnectionConfiguration{name='my-cluster',
> address='',
> > connectorName='artemis', clientFailureCheckPeriod=30000,
> > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > clusterNotificationAttempts=2}
> >
> > 2024-01-11 18:24:14,081 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > 2024-01-11 18:24:14,083 TRACE
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> >
> > java.lang.Exception:trace<ftp://java.lang.Exception/trace>
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4 rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying reconnection attempt 0/0
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18 > > <mailto:connectorFactory<mailto:connectorFactory>
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> >
> > and currentConnectorConfig: TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Connector NettyConnector
> > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using native epoll
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Started EPOLL Netty Connector version 4.1.86.Final to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Remote destination:
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.25:61616<ftp://10.44.0.25/61616>
> >
> >
> > Thanks,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>>
> > Date: Wednesday, December 13, 2023 at 3:23 PM
> > To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > ActiveMQ Artemis ships with an example in the
> > examples/features/ha/replicated-failback directory that demonstrates how
> > this works. I ran this example and it worked fine. I also started up the
> > brokers manually and triggered failover and failback. Again, everything
> > worked fine. My only guess is that there's something configured wrong or
> > perhaps there's an environmental issue that's preventing failback from
> > working properly for you. If you could provide clear steps on how to
> > reproduce this issue I could investigate further.
> >
> >
> > Justin
> >
> > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am having an issue with failback to the master broker. I start the
> > > master broker then the slave broker. I see the replication from the
> > master
> > > to the slave happen. I stop the master/active broker and a failover
> > happens
> > > to the slave as expected. When the master is started a failback to it
> is
> > > expected but does not happen. The following is my ha-policy config.
> > Did I
> > > miss something in the ha-policy configuration? We are using version
> > 2.28.0
> > > and java17.
> > >
> > > Thanks,
> > > Paul
> > >
> > >
> > > Master broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <master>
> > >
> > > <check-for-live-server>true</check-for-live-server>
> > >
> > > </master>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > Slave broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > The following are the logs from the master and slave brokers
> > > HA “master” broker log after restart
> > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > Starting embedded web server
> > > 2023-12-08 20:42:57,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 15, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,627 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3
> > > 2023-12-08 20:42:57,657 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2 > > > [ID=47edf16f<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > [ID=47edf16f>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/10.44.0.18:34508<http://10.44.0.18:34508>]]
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > > activemq-branding plugin
> > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > artemis-plugin plugin
> > > 2023-12-08 20:42:57,766 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 18, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,767 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 20, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > Initialising
> > > hawtio services
> > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > Configuration
> > > will be discovered via system properties
> > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > Hawtio 2.15.0
> > > 2023-12-08 20:42:57,975 INFO
> > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > authentication filter, JAAS realm: "activemq" authorized role(s):
> "admin"
> > > role principal classes:
> > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > servlet is disabled
> > > 2023-12-08 20:42:57,989 INFO
> > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > > property: [key=policyLocation,
> > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > HTTP Server started at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> >
> > >
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > Artemis Jolokia REST API available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08> > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08> > >
> > >
> > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > Console
> > > available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08>
> >
> > >
> > > 20:42:58,455 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl]
> > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d > > > [ID=f85ec51b<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > [ID=f85ec51b>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:49865<http://127.0.0.6:49865>]]
> > > 2023-12-08 20:42:59,524 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858 > > > [ID=7cef23a2<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > [ID=7cef23a2>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:59691<http://127.0.0.6:59691>]]
> > > 2023-12-08 20:42:59,535 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 22, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,536 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,549 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 24, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,550 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,583 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: OK
> > > 2023-12-08 20:42:59,584 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@5f46053b
> > > 2023-12-08 20:42:59,593 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > 2023-12-08 20:42:59,594 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 28, result: OK
> > > 2023-12-08 20:42:59,616 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:59,615 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 31, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 32, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 35, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 39, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:43:08,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256 > > > [ID=1d1d79ce<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > [ID=1d1d79ce>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:60787<http://127.0.0.6:60787>]]
> > > 2023-12-08 20:43:08,464 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7 > > > [ID=8f88c6d5<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > [ID=8f88c6d5>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:40065<http://127.0.0.6:40065>]]
> > >
> > >
> > > HA “slave” broker log.
> > >
> > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > synchronized with live server,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > >
> > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221031: backup announced
> > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221084: Requested 0 quorum votes
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221071: Failing over based on quorum vote results.
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:51,946 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:56,977 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > destroyed
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2023-12-08 20:42:57,445 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6
> > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > > Retrying
> > > 2023-12-08 20:42:57,629 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > > [ID=483170f6<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:55911<http://127.0.0.6:55911>]]
> > > 2023-12-08 20:42:57,630 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > > [ID=4514a2b3<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:60199<http://127.0.0.6:60199>]]
> > > 2023-12-08 20:42:57,648 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > > [ID=fcde4d78<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:53221<http://127.0.0.6:53221>]]
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 29, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 31, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,708 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 33, result: OK
> > > 2023-12-08 20:42:57,709 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@3645037f
> > > 2023-12-08 20:42:57,719 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 35, result: OK
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:57,753 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > 2023-12-08 20:42:57,760 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 38, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, > > > with transaction: null
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 41, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 43, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 44, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 46, result: OK
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 48, result: OK
> > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > >
> > >
> > >
> >
>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by Justin Bertram <jb...@apache.org>.
Anoop, if you wish to unsubscribe from the ActiveMQ users list use the
email address listed on the website [1] (i.e.
users-unsubscribe@activemq.apache.org).
Justin
[1] https://activemq.apache.org/contact
On Fri, Jan 19, 2024 at 11:06 AM Anoop Sikhwal
<an...@infodesk.com.invalid> wrote:
> Unsubscribe
>
> ------------------------------
> *From:* John Lilley <jo...@redpointglobal.com.INVALID>
> *Sent:* Friday, January 19, 2024 10:24:39 PM
> *To:* users@activemq.apache.org <us...@activemq.apache.org>
> *Subject:* RE: Trouble with Replication HA Master/Slave config performing
> failback
>
>
> Hi Paul,
>
>
>
> Thanks for all of this! One thing you said struck me as odd: “You will
> need a separate PVC for both your Primary and standby broker instances.”
>
>
>
> By “separate PVC” do you mean completely different volumes? Or a shared
> volume with a mount in each pod?
>
>
>
> Does this mean that the primary and standby do not communicate anything
> via the lock file? Perhaps I am confused by the name “lock file” which
> implies some kind of concurrency management.
>
>
>
> Instead of two separate PVC volumes, would it be OK to use a shared PVC
> (in Azure, this is an NFS-mounted FileShare volume) but set the primary and
> backup to use different directories within the volume? To save cost and
> complexity.
>
>
>
> john
>
>
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> *From:* Shields, Paul Michael <pa...@hpe.com>
> *Sent:* Friday, January 19, 2024 9:48 AM
> *To:* users@activemq.apache.org
> *Subject:* Re: Trouble with Replication HA Master/Slave config performing
> failback
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Hi John,
>
>
>
> I am using the activemq-artemis-operator from ArtemisCloud.io
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fArtemisCloud.io&c=E,1,TV8NB-Ny1beQqMowEfKkewBSKybapQAV_KnDp6P5OgtMCv0-n88PFErhGYbfjzDTXTShqF8_nA7PcwZSbhhhq9T1SeZtmTI-yE7OpN_U2KevLKqACVtBmbc,&typo=1&ancr_add=1>
> That operator only creates the PVC when I specify persistenceEnabled=true.
> So that is how I create a PVC for each broker instance. I then have to
> customize the broker.xml file to only have the
> <node-manager-lock-directory> on the PVC and the othe Artemis data
> directories on the local pod filesystem.
>
>
>
> If you are deploying directly to k8s without an operator then you could
> have persistenceEnabled=false, but would have to define a PVC that would
> mount the <node-manager-lock-directory> to the broker pod to store the
> server.lock file. You will need a separate PVC for both your Primary and
> standby broker instances.
>
>
>
> Yes, I have attached the broker.xml file to this email. Note that the
> only difference between my primary and standby instance broker.xml files is
> the <ha-policy> definition. So, I will attach the whole broker.xml from my
> Primary and just post the standby <ha-policy> portion.
>
>
>
> Show the PVC mount:
>
> *ncn-m001:~ #* kubectl -n dvs exec -it cray-dvs-mqtt-ss-0 -- /bin/bash
>
> [jboss@cray-dvs-mqtt-ss-0 ~]$ df
>
> Filesystem 1K-blocks Used Available Use% Mounted on
>
> containerd_overlayfs 676576488 219342084 457234404 33% /
>
> tmpfs 65536 0 65536 0% /dev
>
> tmpfs 131773112 0 131773112 0% /sys/fs/cgroup
>
> containerd_overlayfs 676576488 219342084 457234404 33% /etc/hostname
>
> shm 65536 0 65536 0% /dev/shm
>
> /dev/sdc3 187445840 222380 187223460 1% /etc/hosts
>
> /dev/rbd2 1992552 34876 1941292 2%
> /opt/cray-dvs-mqtt/data ß- this is the PVC mount
>
> tmpfs 263443828 4 263443824 1%
> /amq/extra/secrets/cray-dvs-mqtt-props
>
> tmpfs 263443828 12 263443816 1% /run/secrets/
> //kubernetes.io/serviceaccount
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f%2f%2fkubernetes.io%2fserviceaccount&c=E,1,zx-JtrSyBrNbA01-NTG91tZA5WOC-kl6BPtfG3RTuIQykLDhNJnVTq3pNbooNd06f7PobYG4wYRoE_WMe607UGKQe_yDrIGm_N6BvubSj6O0VVM,&typo=1&ancr_add=1>
>
> tmpfs 131773112 0 131773112 0% /proc/acpi
>
> tmpfs 131773112 0 131773112 0% /proc/scsi
>
> tmpfs 131773112 0 131773112 0% /sys/firmware
>
>
>
> Show the lock file on the PVC
>
> [jboss@cray-dvs-mqtt-ss-0 ~]$ od -x
> /opt/cray-dvs-mqtt/data/journal/server.lock
>
> 0000000 3050 fa30 65aa b57a 117b 9eee b25c bf25
>
> 0000020 7183 0096
>
> 0000023
>
> [jboss@cray-dvs-mqtt-ss-0 ~]$ od -x
> /opt/cray-dvs-mqtt/data/journal/serverlock.1
>
> 0000000
>
> [jboss@cray-dvs-mqtt-ss-0 ~]$ od -x
> /opt/cray-dvs-mqtt/data/journal/serverlock.2
>
> 0000000
>
> [jboss@cray-dvs-mqtt-ss-0 ~]$ exit
>
> exit
>
> *ncn-m001:~ #* kubectl -n dvs exec -it cray-dvs-mqtt-ss-1 -- /bin/bash
>
> [jboss@cray-dvs-mqtt-ss-1 ~]$ ls -l /opt/cray-dvs-mqtt/data/journal/
>
> total 4
>
> -rw-r--r-- 1 jboss root 19 Jan 17 22:26 server.lock
>
> -rw-r--r-- 1 jboss root 0 Jan 17 22:25 serverlock.1
>
> -rw-r--r-- 1 jboss root 0 Jan 17 22:25 serverlock.2
>
> [jboss@cray-dvs-mqtt-ss-1 ~]$ od -x
> /opt/cray-dvs-mqtt/data/journal/server.lock
>
> 0000000 3030 fa30 65aa b57a 117b 9eee b25c bf25
>
> 0000020 7183 0096
>
> 0000023
>
> [jboss@cray-dvs-mqtt-ss-1 ~]$
>
>
>
> Primary instance broker.xml file: Note I have logging enable. You
> probably don’t want that in a production environment.
>
>
>
> [jboss@cray-dvs-mqtt-ss-0 ~]$ cat amq-broker/etc/broker.xml
>
> <?xml version='1.0'?>
>
> <!--
>
> Licensed to the Apache Software Foundation (ASF) under one
>
> or more contributor license agreements. See the NOTICE file
>
> distributed with this work for additional information
>
> regarding copyright ownership. The ASF licenses this file
>
> to you under the Apache License, Version 2.0 (the
>
> "License"); you may not use this file except in compliance
>
> with the License. You may obtain a copy of the License at
>
>
>
> http://www.apache.org/licenses/LICENSE-2.0
>
>
>
> Unless required by applicable law or agreed to in writing,
>
> software distributed under the License is distributed on an
>
> "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
>
> KIND, either express or implied. See the License for the
>
> specific language governing permissions and limitations
>
> under the License.
>
> -->
>
>
>
> <configuration xmlns="urn:activemq"
>
> xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
>
> xmlns:xi=http://www.w3.org/2001/XInclude
>
> xsi:schemaLocation="urn:activemq
> /schema/artemis-configuration.xsd">
>
>
>
> <core xmlns="urn:activemq:core" xmlns:xsi=
> http://www.w3.org/2001/XMLSchema-instance
>
> xsi:schemaLocation="urn:activemq:core ">
>
>
>
> <name>amq-broker</name>
>
>
>
> <ha-policy>
>
> <replication>
>
> <master>
>
> <check-for-live-server>true</check-for-live-server>
>
> </master>
>
> </replication>
>
> </ha-policy>
>
>
>
>
>
>
>
> <persistence-enabled>true</persistence-enabled>
>
>
>
> <!-- this could be ASYNCIO, MAPPED, NIO
>
> ASYNCIO: Linux Libaio
>
> MAPPED: mmap files
>
> NIO: Plain Java Files
>
> -->
>
> <journal-type>NIO</journal-type>
>
>
>
>
> <node-manager-lock-directory>/opt/cray-dvs-mqtt/data/journal</node-manager-lock-directory>
>
>
>
> <paging-directory>data/paging</paging-directory>
>
>
>
> <bindings-directory>data/bindings</bindings-directory>
>
>
>
> <journal-directory>data/journal</journal-directory>
>
>
>
>
> <large-messages-directory>data/large-messages</large-messages-directory>
>
>
>
>
>
> <!-- if you want to retain your journal uncomment this following
> configuration.
>
>
>
> This will allow your system to keep 7 days of your data, up to 10G.
> Tweak it accordingly to your use case and capacity.
>
>
>
> it is recommended to use a separate storage unit from the journal
> for performance considerations.
>
>
>
> <journal-retention-directory period="7" unit="DAYS"
> storage-limit="10G">data/retention</journal-retention-directory>
>
>
>
> You can also enable retention by using the argument
> journal-retention on the `artemis create` command -->
>
>
>
>
>
>
>
> <journal-datasync>true</journal-datasync>
>
>
>
> <journal-min-files>2</journal-min-files>
>
>
>
> <journal-pool-files>10</journal-pool-files>
>
>
>
> <journal-device-block-size>4096</journal-device-block-size>
>
>
>
> <journal-file-size>10M</journal-file-size>
>
> <!--
>
> You can verify the network health of a particular NIC by
> specifying the <network-check-NIC> element.
>
> <network-check-NIC>theNicName</network-check-NIC>
>
> -->
>
>
>
> <!--
>
> Use this to use an HTTP server to validate the network
>
> <network-check-URL-list>
> http://www.apache.org</network-check-URL-list> -->
>
>
>
> <!-- <network-check-period>10000</network-check-period> -->
>
> <!-- <network-check-timeout>1000</network-check-timeout> -->
>
>
>
> <!-- this is a comma separated list, no spaces, just DNS or IPs
>
> it should accept IPV6
>
>
>
> Warning: Make sure you understand your network topology as this
> is meant to validate if your network is valid.
>
> Using IPs that could eventually disappear or be
> partially visible may defeat the purpose.
>
> You can use a list of multiple IPs, and if any
> successful ping will make the server OK to continue running -->
>
> <!-- <network-check-list>10.0.0.1</network-check-list> -->
>
>
>
> <!-- use this to customize the ping used for ipv4 addresses -->
>
> <!-- <network-check-ping-command>ping -c 1 -t %d
> %s</network-check-ping-command> -->
>
>
>
> <!-- use this to customize the ping used for ipv6 addresses -->
>
> <!-- <network-check-ping6-command>ping6 -c 1
> %2$s</network-check-ping6-command> -->
>
>
>
>
>
>
>
> <connectors>
>
> <!-- Connector used to be announced through cluster connections
> and notifications -->
>
> <connector
> name="artemis">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616</connector>
>
> </connectors>
>
>
>
>
>
>
>
> <!-- how often we are looking for how many bytes are being used on
> the disk in ms -->
>
> <disk-scan-period>5000</disk-scan-period>
>
>
>
> <!-- once the disk hits this limit the system will block, or close
> the connection in certain protocols
>
> that won't support flow control. -->
>
> <max-disk-usage>90</max-disk-usage>
>
>
>
> <!-- should the broker detect dead locks and other issues -->
>
> <critical-analyzer>true</critical-analyzer>
>
>
>
> <critical-analyzer-timeout>120000</critical-analyzer-timeout>
>
>
>
>
> <critical-analyzer-check-period>60000</critical-analyzer-check-period>
>
>
>
> <critical-analyzer-policy>HALT</critical-analyzer-policy>
>
>
>
>
>
>
>
> <!-- the system will enter into page mode once you hit this limit.
> This is an estimate in bytes of how much the messages are using in memory
>
>
>
> The system will use half of the available memory (-Xmx) by default
> for the global-max-size.
>
> You may specify a different value here if you need to customize it
> to your needs.
>
>
>
> <global-max-size>100Mb</global-max-size> -->
>
>
>
> <!-- the maximum number of messages accepted before entering full
> address mode.
>
> if global-max-size is specified the full address mode will be
> specified by whatever hits it first. -->
>
> <global-max-messages>-1</global-max-messages>
>
>
>
> <acceptors>
>
>
>
> <!-- useEpoll means: it will use Netty epoll if you are on a
> system (Linux) that supports it -->
>
> <!-- amqpCredits: The number of credits sent to AMQP producers -->
>
> <!-- amqpLowCredits: The server will send the # credits specified
> at amqpCredits at this low mark -->
>
> <!-- amqpDuplicateDetection: If you are not using duplicate
> detection, set this to false
>
> as duplicate detection requires
> applicationProperties to be parsed on the server. -->
>
> <!-- amqpMinLargeMessageSize: Determines how many bytes are
> considered large, so we start using files to hold their data.
>
> default: 102400, -1 would mean to
> disable large mesasge control -->
>
>
>
> <!-- Note: If an acceptor needs to be compatible with HornetQ
> and/or Artemis 1.x clients add
>
> "anycastPrefix=jms.queue.;multicastPrefix=jms.topic."
> to the acceptor url.
>
> See https://issues.apache.org/jira/browse/ARTEMIS-1644
> for more information. -->
>
>
>
>
>
> <!-- Acceptor for every supported protocol -->
>
>
>
> <acceptor
> name="dvs">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883?protocols=MQTT;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor><acceptor
> name="scaleDown">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616?protocols=CORE;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor>
>
> </acceptors>
>
>
>
>
>
> <cluster-user>EtBU3LYU</cluster-user>
>
>
>
> <cluster-password>xOKqDzwH</cluster-password>
>
>
>
> <broadcast-groups>
>
> <broadcast-group name="my-broadcast-group">
> <jgroups-file>jgroups-ping.xml</jgroups-file>
> <jgroups-channel>activemq_broadcast_channel</jgroups-channel>
> <connector-ref>artemis</connector-ref> </broadcast-group>
>
> </broadcast-groups>
>
>
>
> <discovery-groups>
>
> <discovery-group name="my-discovery-group">
> <jgroups-file>jgroups-ping.xml</jgroups-file>
> <jgroups-channel>activemq_broadcast_channel</jgroups-channel>
> <refresh-timeout>10000</refresh-timeout> </discovery-group>
>
> </discovery-groups>
>
>
>
> <cluster-connections>
>
> <cluster-connection name="my-cluster">
> <connector-ref>artemis</connector-ref>
> <retry-interval>1000</retry-interval>
> <retry-interval-multiplier>2</retry-interval-multiplier>
> <max-retry-interval>32000</max-retry-interval>
> <initial-connect-attempts>20</initial-connect-attempts>
> <reconnect-attempts>-1</reconnect-attempts>
> <use-duplicate-detection>true</use-duplicate-detection>
> <message-load-balancing>ON_DEMAND</message-load-balancing>
> <max-hops>1</max-hops> <discovery-group-ref
> discovery-group-name="my-discovery-group"/> </cluster-connection>
>
> </cluster-connections>
>
>
>
>
>
> <security-settings>
>
> <security-setting match="#">
>
> <permission type="createNonDurableQueue" roles="admin"/>
>
> <permission type="deleteNonDurableQueue" roles="admin"/>
>
> <permission type="createDurableQueue" roles="admin"/>
>
> <permission type="deleteDurableQueue" roles="admin"/>
>
> <permission type="createAddress" roles="admin"/>
>
> <permission type="deleteAddress" roles="admin"/>
>
> <permission type="consume" roles="admin"/>
>
> <permission type="browse" roles="admin"/>
>
> <permission type="send" roles="admin"/>
>
> <!-- we need this otherwise ./artemis data imp wouldn't work
> -->
>
> <permission type="manage" roles="admin"/>
>
> </security-setting>
>
> </security-settings>
>
>
>
> <address-settings>
>
> <!-- if you define auto-create on certain queues, management has
> to be auto-create -->
>
> <address-setting match="//activemq.management#">
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f%2f%2factivemq.management%23%26quot%3b%26gt%3b&c=E,1,BRroUY8kfUeX80gvrp7t72VVHVGMPASjpLV9yYkz4JVee8GpRfZvI_Kw2UOOUNV2uD0jK5gOPSY2kMhPWHhNzPMAHND9p06dXyXm9NI42fIRLB9HOfs,&typo=1&ancr_add=1>
>
> <dead-letter-address>DLQ</dead-letter-address>
>
> <expiry-address>ExpiryQueue</expiry-address>
>
> <redelivery-delay>0</redelivery-delay>
>
> <!-- with -1 only the global-max-size is in use for limiting
> -->
>
> <max-size-bytes>-1</max-size-bytes>
>
>
> <message-counter-history-day-limit>10</message-counter-history-day-limit>
>
> <address-full-policy>PAGE</address-full-policy>
>
> <auto-create-queues>true</auto-create-queues>
>
> <auto-create-addresses>true</auto-create-addresses>
>
> </address-setting>
>
> <!--default for catch all-->
>
> <address-setting match="#">
>
> <redistribution-delay>0</redistribution-delay>
>
> <dead-letter-address>DLQ</dead-letter-address>
>
> <expiry-address>ExpiryQueue</expiry-address>
>
> <redelivery-delay>0</redelivery-delay>
>
>
>
> <!-- if max-size-bytes and max-size-messages were both
> enabled, the system will enter into paging
>
> based on the first attribute to hits the maximum value -->
>
> <!-- limit for the address in bytes, -1 means unlimited -->
>
> <max-size-bytes>-1</max-size-bytes>
>
>
>
> <!-- limit for the address in messages, -1 means unlimited -->
>
> <max-size-messages>-1</max-size-messages>
>
>
>
> <!-- the size of each file on paging. Notice we keep files in
> memory while they are in use.
>
> Lower this setting if you have too many queues in memory.
> -->
>
> <page-size-bytes>10M</page-size-bytes>
>
>
>
> <!-- limit how many messages are read from paging into the
> Queue. -->
>
> <max-read-page-messages>-1</max-read-page-messages>
>
>
>
> <!-- limit how much memory is read from paging into the Queue.
> -->
>
> <max-read-page-bytes>20M</max-read-page-bytes>
>
>
>
>
> <message-counter-history-day-limit>10</message-counter-history-day-limit>
>
> <address-full-policy>PAGE</address-full-policy>
>
> <auto-create-queues>true</auto-create-queues>
>
> <auto-create-addresses>true</auto-create-addresses>
>
> <auto-delete-queues>false</auto-delete-queues>
>
> <auto-delete-addresses>false</auto-delete-addresses>
>
> </address-setting>
>
> </address-settings>
>
>
>
> <addresses>
>
> <address name="DLQ">
>
> <anycast>
>
> <queue name="DLQ" />
>
> </anycast>
>
> </address>
>
> <address name="ExpiryQueue">
>
> <anycast>
>
> <queue name="ExpiryQueue" />
>
> </anycast>
>
> </address>
>
>
>
> </addresses>
>
>
>
>
>
>
>
> <broker-plugins>
>
> <broker-plugin
> class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
>
> <property key="LOG_ALL_EVENTS" value="true"/>
>
> <property key="LOG_CONNECTION_EVENTS" value="true"/>
>
> <property key="LOG_SESSION_EVENTS" value="true"/>
>
> <property key="LOG_CONSUMER_EVENTS" value="true"/>
>
> <property key="LOG_DELIVERING_EVENTS" value="true"/>
>
> <property key="LOG_SENDING_EVENTS" value="true"/>
>
> <property key="LOG_INTERNAL_EVENTS" value="true"/>
>
> </broker-plugin>
>
> </broker-plugins>
>
>
>
> <metrics> <plugin
> class-name="com.redhat.amq.broker.core.server.metrics.plugins.ArtemisPrometheusMetricsPlugin"/>
> </metrics>
>
> </core>
>
> </configuration>
>
> [jboss@cray-dvs-mqtt-ss-0 ~]$
>
>
>
> Standby instance broker.xml <ha-policy> stanza
>
>
>
> <ha-policy>
>
> <replication>
>
> <slave>
>
> <allow-failback>true</allow-failback>
>
> </slave>
>
> </replication>
>
> </ha-policy>
>
>
>
>
>
> *From: *John Lilley <jo...@redpointglobal.com.INVALID>
> *Date: *Friday, January 19, 2024 at 10:21 AM
> *To: *users@activemq.apache.org <us...@activemq.apache.org>
> *Cc: *Lino Pereira <li...@redpointglobal.com>
> *Subject: *RE: Trouble with Replication HA Master/Slave config performing
> failback
>
> Hi Paul,
>
>
>
> Thanks for the information! It sounds like you have experience with AMQ
> in K8S. Can you share your two broker.xml files with us?
>
>
>
> Lino: do we currently set persistenceEnable=false? I would expect
> there’s no reason for setting it true, since the pod-local storage will
> evaporate with the pod.
>
>
>
> Thanks
>
> John
>
>
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,_MEdc5Qy-vTEnjozH--yv7M1N-lLpEcvjRVTe0TZfe8G_dcxUJzpNCwemRvf2WA7KwCsV-QQ_ejGemmADMP_e4L0WvHHBpjTniOeKgAhT3d5l10,&typo=1>
>
> *John Lilley *
>
> *Data Management Chief Architect, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> *From:* Shields, Paul Michael <pa...@hpe.com>
> *Sent:* Friday, January 19, 2024 8:51 AM
> *To:* users@activemq.apache.org
> *Subject:* Re: Trouble with Replication HA Master/Slave config performing
> failback
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Hi John,
>
>
>
> The <node-manager-lock-directory>. Needs to be specified in both the
> master and slave broker.xml files.
>
>
>
> the nodeUUID which is contained in the server lock file was changing
> between active broker startups. The server.lock file is stored in the
> /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our
> current configuration of the broker has persistenceEnable=false which
> causes any data/broker state to be removed when the broker pod is
> destroyed. Thus a new nodeUUID was being generated on each Primary broker
> startup. When the Primary broker instance starts up it looks for an
> existing server.lock file. When it does not find one it generates a new
> nodeUUID and places it in the server.lock file, otherwise it uses the
> nodeUUID contained within the existing server.lock file. The Primary broker
> then communicates this nodeUUID through the cluster connection to the
> Standby broker instance on startup. The Standby broker instance uses the
> nodeUUID to determine if the Primary broker instance is part of its HA
> pair. The Standby broker instance saved off the original nodeUUID when the
> HA cluster was first started and since the recently communicated nodeUUID
> does not match the nodeUUID that it previously stored the fail back is not
> performed. Once persistence was enabled the Standby broker instance
> performed the fallback to the Primary instance and was no longer in the
> "active" state
>
>
>
> This is what I have observed in our Kubernetes environment. Justin please
> make any corrections to my description of the nodeUUID use.
>
>
>
> Paul
>
> *From: *John Lilley <jo...@redpointglobal.com.INVALID>
> *Date: *Thursday, January 18, 2024 at 5:49 PM
> *To: *users@activemq.apache.org <us...@activemq.apache.org>
> *Subject: *RE: Trouble with Replication HA Master/Slave config performing
> failback
>
> Hi Justin,
>
>
>
> Can you elaborate on the lock file in the <node-manager-lock-directory>?
>
> Does it need to be specified in the broker.xml for both the master and the
> slave?
>
> Does it actually get used as a “lock” in the concurrency sense, to
> coordinate the live/backup election?
>
>
>
> Thanks
>
> john
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,0rYuECbqGwpo9IjFKJrQQrfyzL5V_aEiqSXV25I7DeF4wuQn3I6z1sJCY1NkNJ-bjdpV8nVix_VDU0mK5tgrxuSQaSyrlVLxE5EiTf0Z8G4F&typo=1>
>
> *John Lilley *
>
> *Data Management Chief Architect, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Monday, January 15, 2024 9:56 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Trouble with Replication HA Master/Slave config performing
> failback
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> In the replication case you still need a PV in K8s, but I don't believe
> you need to put the whole journal on the PV (and suffer the performance
> penalty). You should just need to point the node-manager-lock-directory in
> broker.xml to the PV.
>
>
>
>
>
> Justin
>
>
>
> On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hi Justin,
>
> Based on your reply in the provided link, are you saying that even for the
> replication HA case, you still need an external FileShare PV, in K8s, so
> that the nodeID can be persisted between restarts. In this use case, is the
> performance of this FileShare less of a concern than it is for the
> shared-store use case?
>
> Thanks,
> Lino
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Friday, January 12, 2024 4:53 PM
> To: users@activemq.apache.org
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
>
> *** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ***
>
> I sent that email on December 15. You can find it on the web-based mailing
> list interface [1].
>
> Justin
>
> [1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l
>
> On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <
> paul.shields@hpe.com>
> wrote:
>
> > Hi Justin,
> >
> > Forgive me but I don’t remember a journal persistence issue discussion. I
> > did an email search too and did not see that discussion either. Would you
> > please refresh me on the discussion? I may have inadvertently deleted
> that?
> >
> > I do see data being replicated to my production slave/standby instance on
> > initial startup of the two broker instances. Failover works as expected.
> > But when the primary instance starts backup I do not see the active slave
> > replicate to the starting primary and I do not see the active slave
> restart
> > and go to the backup server state.
> >
> > What is the communication sequence when the primary starts up after it
> > quit with the active backup broker? In other words, how is the active
> > backup broker notified to start replication to the newly started primary
> > broker?
> >
> > I ran the replicated-failback example again with TRACE level logging
> > turned on and I do see the attemptFailback=false message in the
> > target/server1/log/artemis.log file at startup, so it must be changed
> later
> > or am I not understanding the purpose of this flag?
> >
> > 2024-01-12 14:40:18,341 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> > discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=localhost, b=null],
> > nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=localhost)
> >
> > From my production slave broker log
> >
> >
> > 024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > Thanks,
> > Paul
> > From: Justin Bertram <jb...@apache.org>
> > Date: Friday, January 12, 2024 at 3:38 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > > Is it possible to use jgroups instead of UDP for the cluster connection
> > in the example?
> >
> > Yes. Check out examples/features/clustered/clustered-jgroups for an
> example
> > configuration. Keep in mind that UDP, JGroups, etc. is *only* used for
> > initial discovery. Once brokers are discovered then normal TCP
> connections
> > are created.
> >
> > Before I continue looking through your logs, etc. can you confirm that
> you
> > resolved the journal persistence issue I described in my previous email?
> If
> > that's not resolved there's no chance for failback to occur.
> >
> >
> > Justin
> >
> >
> >
> > On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> > paul.shields@hpe.com>
> > wrote:
> >
> > > Hi Justin,
> > >
> > > I tried the example and it works. I turned up logging to get a success
> > > path through the code to maybe see where things get de-railed. Is it
> > > possible to use jgroups instead of UDP for the cluster connection in
> the
> > > example? That would give me a better success signature.
> > >
> > > My standby instance has this in the broker.xml which is the same as in
> > the
> > > example:
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > I am seeing this series of log messages that suggests that failback is
> > not
> > > enabled. Is the attemptFailback=false the default at this point in
> > startup
> > > and it is changed later or it there something gone wrong at this point?
> > >
> > > 2024-01-11 18:24:14,066 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.Topology]
> Topology@7e0a913c
> > [owner=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > >
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > > QuorumManager(server=null) about node up =
> > > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > > Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null]
> > >
> > > 2024-01-11 18:24:14,067 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.Topology]
> Topology@7e0a913c
> > [owner=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > >
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > >
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> > > <mailto:
> <%0b>> >
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> > >
> > > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > > Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null]
> > >
> > > 2024-01-11 18:24:14,067 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > ClientSessionFactoryImpl received backup update for live/backup pair =
> > > TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > > / null but it didn't belong to TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > >
> > > 2024-01-11 18:24:14,067 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Cluster Connected
> > >
> > > 2024-01-11 18:24:14,068 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Starting backup manager
> > >
> > > 2024-01-11 18:24:14,069 DEBUG
> > > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > > backup config ClusterConnectionConfiguration{name='my-cluster',
> > address='',
> > > connectorName='artemis', clientFailureCheckPeriod=30000,
> > > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > > maxRetryInterval=32000, initialConnectAttempts=20,
> reconnectAttempts=-1,
> > > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > > staticConnectors=[], discoveryGroupName='my-discovery-group',
> maxHops=1,
> > > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > > clusterNotificationAttempts=2}
> > >
> > > 2024-01-11 18:24:14,081 DEBUG
> > > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > > BackupManager connecting to DiscoveryBackupConnector
> > > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> > >
> > > 2024-01-11 18:24:14,082 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Starting Backup Server
> > >
> > > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Setting server state as started
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > looking up the node through nodeLocator.locateNode()
> > >
> > > 2024-01-11 18:24:14,083 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Connecting towards a possible live, connection
> > > information=Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> > >
> > > 2024-01-11 18:24:14,083 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Calling
> > >
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> > >
> > > java.lang.Exception:trace <ftp://java.lang.exception/trace>
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0
> <ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0
> <ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0
> <ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0
> <ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0
> <ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0
> <ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4
> rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0
> <ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>
> (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0
> <ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > Trying reconnection attempt 0/0
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > Trying to connect with
> > >
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> > > <mailto:connectorFactory
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> > >
> > > and currentConnectorConfig: TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Connector NettyConnector
> > > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> > useServlet=false,
> > > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > > using native epoll
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Started EPOLL Netty Connector version 4.1.86.Final to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> <ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Remote destination:
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.25:61616 <ftp://10.44.0.25/61616>
> > >
> > >
> > > Thanks,
> > > Paul
> > >
> > > From: Justin Bertram <jb...@apache.org>
> > > Date: Wednesday, December 13, 2023 at 3:23 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > > failback
> > > ActiveMQ Artemis ships with an example in the
> > > examples/features/ha/replicated-failback directory that demonstrates
> how
> > > this works. I ran this example and it worked fine. I also started up
> the
> > > brokers manually and triggered failover and failback. Again, everything
> > > worked fine. My only guess is that there's something configured wrong
> or
> > > perhaps there's an environmental issue that's preventing failback from
> > > working properly for you. If you could provide clear steps on how to
> > > reproduce this issue I could investigate further.
> > >
> > >
> > > Justin
> > >
> > > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> > paul.shields@hpe.com
> > > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I am having an issue with failback to the master broker. I start the
> > > > master broker then the slave broker. I see the replication from the
> > > master
> > > > to the slave happen. I stop the master/active broker and a failover
> > > happens
> > > > to the slave as expected. When the master is started a failback to it
> > is
> > > > expected but does not happen. The following is my ha-policy config.
> > > Did I
> > > > miss something in the ha-policy configuration? We are using version
> > > 2.28.0
> > > > and java17.
> > > >
> > > > Thanks,
> > > > Paul
> > > >
> > > >
> > > > Master broker.xml config
> > > >
> > > > <ha-policy>
> > > >
> > > > <replication>
> > > >
> > > > <master>
> > > >
> > > > <check-for-live-server>true</check-for-live-server>
> > > >
> > > > </master>
> > > >
> > > > </replication>
> > > >
> > > > </ha-policy>
> > > >
> > > > Slave broker.xml config
> > > >
> > > > <ha-policy>
> > > >
> > > > <replication>
> > > >
> > > > <slave>
> > > >
> > > > <allow-failback>true</allow-failback>
> > > >
> > > > </slave>
> > > >
> > > > </replication>
> > > >
> > > > </ha-policy>
> > > >
> > > > The following are the logs from the master and slave brokers
> > > > HA “master” broker log after restart
> > > > 2023-12-08 20:42:57,509 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883
> <ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> > for
> > > > protocols [MQTT]
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > >
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> <ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> > for
> > > > protocols [CORE]
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221007: Server is now live
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > > Starting embedded web server
> > > > 2023-12-08 20:42:57,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 15, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,627 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3
> > > > 2023-12-08 20:42:57,657 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > > > [ID=47edf16f<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > > [ID=47edf16f>,
> > > > local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
> > > > 2023-12-08 20:42:57,720 INFO
> > > > [org.apache.activemq.hawtio.branding.PluginContextListener]
> Initialized
> > > > activemq-branding plugin
> > > > 2023-12-08 20:42:57,737 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is connected
> > > > 2023-12-08 20:42:57,755 INFO
> > > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > > artemis-plugin plugin
> > > > 2023-12-08 20:42:57,766 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 18, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,767 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 20, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > > Initialising
> > > > hawtio services
> > > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > > Configuration
> > > > will be discovered via system properties
> > > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > > Hawtio 2.15.0
> > > > 2023-12-08 20:42:57,975 INFO
> > > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > > authentication filter, JAAS realm: "activemq" authorized role(s):
> > "admin"
> > > > role principal classes:
> > > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > > servlet is disabled
> > > > 2023-12-08 20:42:57,989 INFO
> > > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia
> overridden
> > > > property: [key=policyLocation,
> > > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > > HTTP Server started at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> > >
> > > >
> > > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > > Artemis Jolokia REST API available at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> > >
> > > >
> > > > 2023-12-08
> > > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> > >
> > > >
> > > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > > Console
> > > > available at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> > >
> > > >
> > > > 2023-12-08
> > > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> > >
> > > >
> > > > 20:42:58,455 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl]
> > > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > > > [ID=f85ec51b<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > > [ID=f85ec51b>,
> > > > local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
> > > > 2023-12-08 20:42:59,524 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > > > [ID=7cef23a2<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > > [ID=7cef23a2>,
> > > > local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
> > > > 2023-12-08 20:42:59,535 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 22, result: NO_BINDINGS
> > > > 2023-12-08 20:42:59,536 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> > connectionID:
> > > > 7cef23a2
> > > > 2023-12-08 20:42:59,549 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 24, result: NO_BINDINGS
> > > > 2023-12-08 20:42:59,550 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> > connectionID:
> > > > 7cef23a2
> > > > 2023-12-08 20:42:59,583 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 26, result: OK
> > > > 2023-12-08 20:42:59,584 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=true]@5f46053b
> > > > 2023-12-08 20:42:59,593 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> > created
> > > > consumer with ID: 0, with session name:
> > > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > > 2023-12-08 20:42:59,594 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 28, result: OK
> > > > 2023-12-08 20:42:59,616 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > > 2023-12-08 20:42:59,615 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[
> *AMQ*RESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > > 2023-12-08 20:42:59,621 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 31, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,621 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[
> *AMQ*RoutingName=ExpiryQueue,*AMQ*Distance=0,*AMQ*Address=ExpiryQueue,
> *AMQ*NotifType=BINDING_ADDED,*AMQ*Binding_ID=7,*AMQ*
> FilterString=NULL-value,*AMQ*NotifTimestamp=1702068179613,*AMQ*ClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
> > > > with transaction: null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 32, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[
> *AMQ*RoutingName=DLQ,*AMQ*Distance=0,*AMQ*Address=DLQ,*AMQ*
> NotifType=BINDING_ADDED,*AMQ*Binding_ID=3,*AMQ*FilterString=NULL-value,
> *AMQ*NotifTimestamp=1702068179614,*AMQ*ClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
> > > > with transaction: null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 35, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[
> *AMQ*RESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction:
> null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 39, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:43:08,447 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > > > [ID=1d1d79ce<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > > [ID=1d1d79ce>,
> > > > local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
> > > > 2023-12-08 20:43:08,464 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > > > [ID=8f88c6d5<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > > [ID=8f88c6d5>,
> > > > local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
> > > >
> > > >
> > > > HA “slave” broker log.
> > > >
> > > > 2023-12-08 20:37:58,374 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0
> [null]
> > > > started, waiting live to fail before it gets active
> > > >
> > > > 2023-12-08 20:37:59,020 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > > synchronized with live server,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > > >
> > > > 2023-12-08 20:38:00,032 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221031: backup announced
> > > > 2023-12-08 20:42:27,415 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221084: Requested 0 quorum votes
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221071: Failing over based on quorum vote results.
> > > > 2023-12-08 20:42:27,431 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212037: Connection failure to
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > > 10.44.0.20:61616 <ftp://10.44.0.20/61616> has been detected:
> AMQ219015: The connection was
> > > > disconnected because of server shutdown [code=DISCONNECTED]
> > > > 2023-12-08 20:42:27,431 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212037: Connection failure to
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > > 10.44.0.20:61616 <ftp://10.44.0.20/61616> has been detected:
> AMQ219015: The connection was
> > > > disconnected because of server shutdown [code=DISCONNECTED]
> > > > 2023-12-08 20:42:27,434 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > > 2023-12-08 20:42:27,574 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > > 2023-12-08 20:42:27,575 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > > 2023-12-08 20:42:27,578 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > > 2023-12-08 20:42:27,578 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > > 2023-12-08 20:42:27,611 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221007: Server is now live
> > > > 2023-12-08 20:42:27,699 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883
> <ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> > for
> > > > protocols [MQTT]
> > > > 2023-12-08 20:42:27,717 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > >
> cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> <ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> > for
> > > > protocols [CORE]
> > > > 2023-12-08 20:42:51,946 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > > [ID=df357d0e<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > [ID=df357d0e>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > > > 2023-12-08 20:42:56,977 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > > destroyed
> > > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > > [ID=df357d0e<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > [ID=df357d0e>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > > > 2023-12-08 20:42:57,443 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212004: Failed to connect to server.
> > > > 2023-12-08 20:42:57,445 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 26, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,447 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6
> > > > 2023-12-08 20:42:57,512 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > > failure detected. Unblocking a blocking call that will never get a
> > > response
> > > > 2023-12-08 20:42:57,513 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is unable to connect to
> destination.
> > > > Retrying
> > > > 2023-12-08 20:42:57,629 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > > > [ID=483170f6<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > > [ID=483170f6>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
> > > > 2023-12-08 20:42:57,630 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > > > [ID=4514a2b3<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > > [ID=4514a2b3>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
> > > > 2023-12-08 20:42:57,648 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > > > [ID=fcde4d78<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > > [ID=fcde4d78>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
> > > > 2023-12-08 20:42:57,663 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 29, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,663 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> > connectionID:
> > > > fcde4d78
> > > > 2023-12-08 20:42:57,679 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 31, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,679 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> > connectionID:
> > > > fcde4d78
> > > > 2023-12-08 20:42:57,708 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 33, result: OK
> > > > 2023-12-08 20:42:57,709 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=true]@3645037f
> > > > 2023-12-08 20:42:57,719 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> > created
> > > > consumer with ID: 0, with session name:
> > > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > > 2023-12-08 20:42:57,720 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 35, result: OK
> > > > 2023-12-08 20:42:57,755 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > > 2023-12-08 20:42:57,753 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[
> *AMQ*RESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > > 2023-12-08 20:42:57,760 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 38, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,761 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[
> *AMQ*RoutingName=DLQ,*AMQ*Distance=0,*AMQ*Address=DLQ,*AMQ*
> NotifType=BINDING_ADDED,*AMQ*Binding_ID=3,*AMQ*FilterString=NULL-value,
> *AMQ*NotifTimestamp=1702068177753,*AMQ*ClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
> > > > with transaction: null
> > > > 2023-12-08 20:42:57,761 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 41, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[
> *AMQ*RoutingName=ExpiryQueue,*AMQ*Distance=0,*AMQ*Address=ExpiryQueue,
> *AMQ*NotifType=BINDING_ADDED,*AMQ*Binding_ID=7,*AMQ*
> FilterString=NULL-value,*AMQ*NotifTimestamp=1702068177753,*AMQ*ClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
> > > > with transaction: null
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 43, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[
> *AMQ*RESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction:
> null
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 44, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:59,607 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is connected
> > > > 2023-12-08 20:42:59,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 46, result: OK
> > > > 2023-12-08 20:42:59,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 48, result: OK
> > > > 2023-12-08 20:43:07,437 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > > failure detected. Unblocking a blocking call that will never get a
> > > response
> > > >
> > > >
> > > >
> > >
> >
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by Anoop Sikhwal <an...@infodesk.com.INVALID>.
Unsubscribe
________________________________
From: John Lilley <jo...@redpointglobal.com.INVALID>
Sent: Friday, January 19, 2024 10:24:39 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Paul,
Thanks for all of this! One thing you said struck me as odd: “You will need a separate PVC for both your Primary and standby broker instances.”
By “separate PVC” do you mean completely different volumes? Or a shared volume with a mount in each pod?
Does this mean that the primary and standby do not communicate anything via the lock file? Perhaps I am confused by the name “lock file” which implies some kind of concurrency management.
Instead of two separate PVC volumes, would it be OK to use a shared PVC (in Azure, this is an NFS-mounted FileShare volume) but set the primary and backup to use different directories within the volume? To save cost and complexity.
john
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Shields, Paul Michael <pa...@hpe.com>
Sent: Friday, January 19, 2024 9:48 AM
To: users@activemq.apache.org
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hi John,
I am using the activemq-artemis-operator from ArtemisCloud.io<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fArtemisCloud.io&c=E,1,TV8NB-Ny1beQqMowEfKkewBSKybapQAV_KnDp6P5OgtMCv0-n88PFErhGYbfjzDTXTShqF8_nA7PcwZSbhhhq9T1SeZtmTI-yE7OpN_U2KevLKqACVtBmbc,&typo=1&ancr_add=1> That operator only creates the PVC when I specify persistenceEnabled=true. So that is how I create a PVC for each broker instance. I then have to customize the broker.xml file to only have the <node-manager-lock-directory> on the PVC and the othe Artemis data directories on the local pod filesystem.
If you are deploying directly to k8s without an operator then you could have persistenceEnabled=false, but would have to define a PVC that would mount the <node-manager-lock-directory> to the broker pod to store the server.lock file. You will need a separate PVC for both your Primary and standby broker instances.
Yes, I have attached the broker.xml file to this email. Note that the only difference between my primary and standby instance broker.xml files is the <ha-policy> definition. So, I will attach the whole broker.xml from my Primary and just post the standby <ha-policy> portion.
Show the PVC mount:
ncn-m001:~ # kubectl -n dvs exec -it cray-dvs-mqtt-ss-0 -- /bin/bash
[jboss@cray-dvs-mqtt-ss-0 ~]$ df
Filesystem 1K-blocks Used Available Use% Mounted on
containerd_overlayfs 676576488 219342084 457234404 33% /
tmpfs 65536 0 65536 0% /dev
tmpfs 131773112 0 131773112 0% /sys/fs/cgroup
containerd_overlayfs 676576488 219342084 457234404 33% /etc/hostname
shm 65536 0 65536 0% /dev/shm
/dev/sdc3 187445840 222380 187223460 1% /etc/hosts
/dev/rbd2 1992552 34876 1941292 2% /opt/cray-dvs-mqtt/data <--- this is the PVC mount
tmpfs 263443828 4 263443824 1% /amq/extra/secrets/cray-dvs-mqtt-props
tmpfs 263443828 12 263443816 1% /run/secrets///kubernetes.io/serviceaccount<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f%2f%2fkubernetes.io%2fserviceaccount&c=E,1,zx-JtrSyBrNbA01-NTG91tZA5WOC-kl6BPtfG3RTuIQykLDhNJnVTq3pNbooNd06f7PobYG4wYRoE_WMe607UGKQe_yDrIGm_N6BvubSj6O0VVM,&typo=1&ancr_add=1>
tmpfs 131773112 0 131773112 0% /proc/acpi
tmpfs 131773112 0 131773112 0% /proc/scsi
tmpfs 131773112 0 131773112 0% /sys/firmware
Show the lock file on the PVC
[jboss@cray-dvs-mqtt-ss-0 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/server.lock
0000000 3050 fa30 65aa b57a 117b 9eee b25c bf25
0000020 7183 0096
0000023
[jboss@cray-dvs-mqtt-ss-0 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/serverlock.1
0000000
[jboss@cray-dvs-mqtt-ss-0 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/serverlock.2
0000000
[jboss@cray-dvs-mqtt-ss-0 ~]$ exit
exit
ncn-m001:~ # kubectl -n dvs exec -it cray-dvs-mqtt-ss-1 -- /bin/bash
[jboss@cray-dvs-mqtt-ss-1 ~]$ ls -l /opt/cray-dvs-mqtt/data/journal/
total 4
-rw-r--r-- 1 jboss root 19 Jan 17 22:26 server.lock
-rw-r--r-- 1 jboss root 0 Jan 17 22:25 serverlock.1
-rw-r--r-- 1 jboss root 0 Jan 17 22:25 serverlock.2
[jboss@cray-dvs-mqtt-ss-1 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/server.lock
0000000 3030 fa30 65aa b57a 117b 9eee b25c bf25
0000020 7183 0096
0000023
[jboss@cray-dvs-mqtt-ss-1 ~]$
Primary instance broker.xml file: Note I have logging enable. You probably don’t want that in a production environment.
[jboss@cray-dvs-mqtt-ss-0 ~]$ cat amq-broker/etc/broker.xml
<?xml version='1.0'?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<configuration xmlns="urn:activemq"
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
xmlns:xi=http://www.w3.org/2001/XInclude
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
xsi:schemaLocation="urn:activemq:core ">
<name>amq-broker</name>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<persistence-enabled>true</persistence-enabled>
<!-- this could be ASYNCIO, MAPPED, NIO
ASYNCIO: Linux Libaio
MAPPED: mmap files
NIO: Plain Java Files
-->
<journal-type>NIO</journal-type>
<node-manager-lock-directory>/opt/cray-dvs-mqtt/data/journal</node-manager-lock-directory>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<!-- if you want to retain your journal uncomment this following configuration.
This will allow your system to keep 7 days of your data, up to 10G. Tweak it accordingly to your use case and capacity.
it is recommended to use a separate storage unit from the journal for performance considerations.
<journal-retention-directory period="7" unit="DAYS" storage-limit="10G">data/retention</journal-retention-directory>
You can also enable retention by using the argument journal-retention on the `artemis create` command -->
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<!--
You can verify the network health of a particular NIC by specifying the <network-check-NIC> element.
<network-check-NIC>theNicName</network-check-NIC>
-->
<!--
Use this to use an HTTP server to validate the network
<network-check-URL-list>http://www.apache.org</network-check-URL-list<http://www.apache.org%3c/network-check-URL-list>> -->
<!-- <network-check-period>10000</network-check-period> -->
<!-- <network-check-timeout>1000</network-check-timeout> -->
<!-- this is a comma separated list, no spaces, just DNS or IPs
it should accept IPV6
Warning: Make sure you understand your network topology as this is meant to validate if your network is valid.
Using IPs that could eventually disappear or be partially visible may defeat the purpose.
You can use a list of multiple IPs, and if any successful ping will make the server OK to continue running -->
<!-- <network-check-list>10.0.0.1</network-check-list> -->
<!-- use this to customize the ping used for ipv4 addresses -->
<!-- <network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command> -->
<!-- use this to customize the ping used for ipv6 addresses -->
<!-- <network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command> -->
<connectors>
<!-- Connector used to be announced through cluster connections and notifications -->
<connector name="artemis">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616</connector>
</connectors>
<!-- how often we are looking for how many bytes are being used on the disk in ms -->
<disk-scan-period>5000</disk-scan-period>
<!-- once the disk hits this limit the system will block, or close the connection in certain protocols
that won't support flow control. -->
<max-disk-usage>90</max-disk-usage>
<!-- should the broker detect dead locks and other issues -->
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<!-- the system will enter into page mode once you hit this limit. This is an estimate in bytes of how much the messages are using in memory
The system will use half of the available memory (-Xmx) by default for the global-max-size.
You may specify a different value here if you need to customize it to your needs.
<global-max-size>100Mb</global-max-size> -->
<!-- the maximum number of messages accepted before entering full address mode.
if global-max-size is specified the full address mode will be specified by whatever hits it first. -->
<global-max-messages>-1</global-max-messages>
<acceptors>
<!-- useEpoll means: it will use Netty epoll if you are on a system (Linux) that supports it -->
<!-- amqpCredits: The number of credits sent to AMQP producers -->
<!-- amqpLowCredits: The server will send the # credits specified at amqpCredits at this low mark -->
<!-- amqpDuplicateDetection: If you are not using duplicate detection, set this to false
as duplicate detection requires applicationProperties to be parsed on the server. -->
<!-- amqpMinLargeMessageSize: Determines how many bytes are considered large, so we start using files to hold their data.
default: 102400, -1 would mean to disable large mesasge control -->
<!-- Note: If an acceptor needs to be compatible with HornetQ and/or Artemis 1.x clients add
"anycastPrefix=jms.queue.;multicastPrefix=jms.topic." to the acceptor url.
See https://issues.apache.org/jira/browse/ARTEMIS-1644 for more information. -->
<!-- Acceptor for every supported protocol -->
<acceptor name="dvs">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883?protocols=MQTT;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor><acceptor name="scaleDown">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616?protocols=CORE;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor>
</acceptors>
<cluster-user>EtBU3LYU</cluster-user>
<cluster-password>xOKqDzwH</cluster-password>
<broadcast-groups>
<broadcast-group name="my-broadcast-group"> <jgroups-file>jgroups-ping.xml</jgroups-file> <jgroups-channel>activemq_broadcast_channel</jgroups-channel> <connector-ref>artemis</connector-ref> </broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group"> <jgroups-file>jgroups-ping.xml</jgroups-file> <jgroups-channel>activemq_broadcast_channel</jgroups-channel> <refresh-timeout>10000</refresh-timeout> </discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster"> <connector-ref>artemis</connector-ref> <retry-interval>1000</retry-interval> <retry-interval-multiplier>2</retry-interval-multiplier> <max-retry-interval>32000</max-retry-interval> <initial-connect-attempts>20</initial-connect-attempts> <reconnect-attempts>-1</reconnect-attempts> <use-duplicate-detection>true</use-duplicate-detection> <message-load-balancing>ON_DEMAND</message-load-balancing> <max-hops>1</max-hops> <discovery-group-ref discovery-group-name="my-discovery-group"/> </cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="admin"/>
<permission type="deleteNonDurableQueue" roles="admin"/>
<permission type="createDurableQueue" roles="admin"/>
<permission type="deleteDurableQueue" roles="admin"/>
<permission type="createAddress" roles="admin"/>
<permission type="deleteAddress" roles="admin"/>
<permission type="consume" roles="admin"/>
<permission type="browse" roles="admin"/>
<permission type="send" roles="admin"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="admin"/>
</security-setting>
</security-settings>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="//activemq.management#"><https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f%2f%2factivemq.management%23%26quot%3b%26gt%3b&c=E,1,BRroUY8kfUeX80gvrp7t72VVHVGMPASjpLV9yYkz4JVee8GpRfZvI_Kw2UOOUNV2uD0jK5gOPSY2kMhPWHhNzPMAHND9p06dXyXm9NI42fIRLB9HOfs,&typo=1&ancr_add=1>
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<redistribution-delay>0</redistribution-delay>
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- if max-size-bytes and max-size-messages were both enabled, the system will enter into paging
based on the first attribute to hits the maximum value -->
<!-- limit for the address in bytes, -1 means unlimited -->
<max-size-bytes>-1</max-size-bytes>
<!-- limit for the address in messages, -1 means unlimited -->
<max-size-messages>-1</max-size-messages>
<!-- the size of each file on paging. Notice we keep files in memory while they are in use.
Lower this setting if you have too many queues in memory. -->
<page-size-bytes>10M</page-size-bytes>
<!-- limit how many messages are read from paging into the Queue. -->
<max-read-page-messages>-1</max-read-page-messages>
<!-- limit how much memory is read from paging into the Queue. -->
<max-read-page-bytes>20M</max-read-page-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-delete-queues>false</auto-delete-queues>
<auto-delete-addresses>false</auto-delete-addresses>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
<broker-plugins>
<broker-plugin class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
<property key="LOG_ALL_EVENTS" value="true"/>
<property key="LOG_CONNECTION_EVENTS" value="true"/>
<property key="LOG_SESSION_EVENTS" value="true"/>
<property key="LOG_CONSUMER_EVENTS" value="true"/>
<property key="LOG_DELIVERING_EVENTS" value="true"/>
<property key="LOG_SENDING_EVENTS" value="true"/>
<property key="LOG_INTERNAL_EVENTS" value="true"/>
</broker-plugin>
</broker-plugins>
<metrics> <plugin class-name="com.redhat.amq.broker.core.server.metrics.plugins.ArtemisPrometheusMetricsPlugin"/> </metrics>
</core>
</configuration>
[jboss@cray-dvs-mqtt-ss-0 ~]$
Standby instance broker.xml <ha-policy> stanza
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Date: Friday, January 19, 2024 at 10:21 AM
To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
Cc: Lino Pereira <li...@redpointglobal.com>>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Paul,
Thanks for the information! It sounds like you have experience with AMQ in K8S. Can you share your two broker.xml files with us?
Lino: do we currently set persistenceEnable=false? I would expect there’s no reason for setting it true, since the pod-local storage will evaporate with the pod.
Thanks
John
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,_MEdc5Qy-vTEnjozH--yv7M1N-lLpEcvjRVTe0TZfe8G_dcxUJzpNCwemRvf2WA7KwCsV-QQ_ejGemmADMP_e4L0WvHHBpjTniOeKgAhT3d5l10,&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Shields, Paul Michael <pa...@hpe.com>>
Sent: Friday, January 19, 2024 8:51 AM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hi John,
The <node-manager-lock-directory>. Needs to be specified in both the master and slave broker.xml files.
the nodeUUID which is contained in the server lock file was changing between active broker startups. The server.lock file is stored in the /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our current configuration of the broker has persistenceEnable=false which causes any data/broker state to be removed when the broker pod is destroyed. Thus a new nodeUUID was being generated on each Primary broker startup. When the Primary broker instance starts up it looks for an existing server.lock file. When it does not find one it generates a new nodeUUID and places it in the server.lock file, otherwise it uses the nodeUUID contained within the existing server.lock file. The Primary broker then communicates this nodeUUID through the cluster connection to the Standby broker instance on startup. The Standby broker instance uses the nodeUUID to determine if the Primary broker instance is part of its HA pair. The Standby broker instance saved off the original nodeUUID when the HA cluster was first started and since the recently communicated nodeUUID does not match the nodeUUID that it previously stored the fail back is not performed. Once persistence was enabled the Standby broker instance performed the fallback to the Primary instance and was no longer in the "active" state
This is what I have observed in our Kubernetes environment. Justin please make any corrections to my description of the nodeUUID use.
Paul
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Date: Thursday, January 18, 2024 at 5:49 PM
To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Justin,
Can you elaborate on the lock file in the <node-manager-lock-directory>?
Does it need to be specified in the broker.xml for both the master and the slave?
Does it actually get used as a “lock” in the concurrency sense, to coordinate the live/backup election?
Thanks
john
[rg]<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,0rYuECbqGwpo9IjFKJrQQrfyzL5V_aEiqSXV25I7DeF4wuQn3I6z1sJCY1NkNJ-bjdpV8nVix_VDU0mK5tgrxuSQaSyrlVLxE5EiTf0Z8G4F&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Monday, January 15, 2024 9:56 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
In the replication case you still need a PV in K8s, but I don't believe you need to put the whole journal on the PV (and suffer the performance penalty). You should just need to point the node-manager-lock-directory in broker.xml to the PV.
Justin
On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
Based on your reply in the provided link, are you saying that even for the replication HA case, you still need an external FileShare PV, in K8s, so that the nodeID can be persisted between restarts. In this use case, is the performance of this FileShare less of a concern than it is for the shared-store use case?
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
-----Original Message-----
From: Justin Bertram <jb...@apache.org>>
Sent: Friday, January 12, 2024 4:53 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
I sent that email on December 15. You can find it on the web-based mailing
list interface [1].
Justin
[1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l
On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <pa...@hpe.com>>
wrote:
> Hi Justin,
>
> Forgive me but I don’t remember a journal persistence issue discussion. I
> did an email search too and did not see that discussion either. Would you
> please refresh me on the discussion? I may have inadvertently deleted that?
>
> I do see data being replicated to my production slave/standby instance on
> initial startup of the two broker instances. Failover works as expected.
> But when the primary instance starts backup I do not see the active slave
> replicate to the starting primary and I do not see the active slave restart
> and go to the backup server state.
>
> What is the communication sequence when the primary starts up after it
> quit with the active backup broker? In other words, how is the active
> backup broker notified to start replication to the newly started primary
> broker?
>
> I ran the replicated-failback example again with TRACE level logging
> turned on and I do see the attemptFailback=false message in the
> target/server1/log/artemis.log file at startup, so it must be changed later
> or am I not understanding the purpose of this flag?
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> discoveryInitialWaitTimeout=10000}]
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost, b=null],
> nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost)
>
> From my production slave broker log
>
>
> 024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> Thanks,
> Paul
> From: Justin Bertram <jb...@apache.org>>
> Date: Friday, January 12, 2024 at 3:38 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > Is it possible to use jgroups instead of UDP for the cluster connection
> in the example?
>
> Yes. Check out examples/features/clustered/clustered-jgroups for an example
> configuration. Keep in mind that UDP, JGroups, etc. is only used for
> initial discovery. Once brokers are discovered then normal TCP connections
> are created.
>
> Before I continue looking through your logs, etc. can you confirm that you
> resolved the journal persistence issue I described in my previous email? If
> that's not resolved there's no chance for failback to occur.
>
>
> Justin
>
>
>
> On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>>
> wrote:
>
> > Hi Justin,
> >
> > I tried the example and it works. I turned up logging to get a success
> > path through the code to maybe see where things get de-railed. Is it
> > possible to use jgroups instead of UDP for the cluster connection in the
> > example? That would give me a better success signature.
> >
> > My standby instance has this in the broker.xml which is the same as in
> the
> > example:
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > I am seeing this series of log messages that suggests that failback is
> not
> > enabled. Is the attemptFailback=false the default at this point in
> startup
> > and it is changed later or it there something gone wrong at this point?
> >
> > 2024-01-11 18:24:14,066 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > QuorumManager(server=null) about node up =
> > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> > <mailto:
<mailto:%0b>> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> >
> > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > ClientSessionFactoryImpl received backup update for live/backup pair =
> > TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > / null but it didn't belong to TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,067 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Cluster Connected
> >
> > 2024-01-11 18:24:14,068 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting backup manager
> >
> > 2024-01-11 18:24:14,069 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > backup config ClusterConnectionConfiguration{name='my-cluster',
> address='',
> > connectorName='artemis', clientFailureCheckPeriod=30000,
> > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > clusterNotificationAttempts=2}
> >
> > 2024-01-11 18:24:14,081 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > 2024-01-11 18:24:14,083 TRACE
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> >
> > java.lang.Exception:trace<ftp://java.lang.exception/trace>
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4 rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying reconnection attempt 0/0
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18> > > <mailto:connectorFactory<mailto:connectorFactory>
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18>
> >
> > and currentConnectorConfig: TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Connector NettyConnector
> > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using native epoll
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Started EPOLL Netty Connector version 4.1.86.Final to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Remote destination:
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.25:61616<ftp://10.44.0.25/61616>
> >
> >
> > Thanks,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>>
> > Date: Wednesday, December 13, 2023 at 3:23 PM
> > To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > ActiveMQ Artemis ships with an example in the
> > examples/features/ha/replicated-failback directory that demonstrates how
> > this works. I ran this example and it worked fine. I also started up the
> > brokers manually and triggered failover and failback. Again, everything
> > worked fine. My only guess is that there's something configured wrong or
> > perhaps there's an environmental issue that's preventing failback from
> > working properly for you. If you could provide clear steps on how to
> > reproduce this issue I could investigate further.
> >
> >
> > Justin
> >
> > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am having an issue with failback to the master broker. I start the
> > > master broker then the slave broker. I see the replication from the
> > master
> > > to the slave happen. I stop the master/active broker and a failover
> > happens
> > > to the slave as expected. When the master is started a failback to it
> is
> > > expected but does not happen. The following is my ha-policy config.
> > Did I
> > > miss something in the ha-policy configuration? We are using version
> > 2.28.0
> > > and java17.
> > >
> > > Thanks,
> > > Paul
> > >
> > >
> > > Master broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <master>
> > >
> > > <check-for-live-server>true</check-for-live-server>
> > >
> > > </master>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > Slave broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > The following are the logs from the master and slave brokers
> > > HA “master” broker log after restart
> > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > Starting embedded web server
> > > 2023-12-08 20:42:57,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 15, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,627 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3
> > > 2023-12-08 20:42:57,657 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2> > > > [ID=47edf16f<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2>
> > [ID=47edf16f>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616/>, remote=/10.44.0.18:34508<http://10.44.0.18:34508/>]]
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > > activemq-branding plugin
> > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > artemis-plugin plugin
> > > 2023-12-08 20:42:57,766 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 18, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,767 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 20, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > Initialising
> > > hawtio services
> > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > Configuration
> > > will be discovered via system properties
> > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > Hawtio 2.15.0
> > > 2023-12-08 20:42:57,975 INFO
> > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > authentication filter, JAAS realm: "activemq" authorized role(s):
> "admin"
> > > role principal classes:
> > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > servlet is disabled
> > > 2023-12-08 20:42:57,989 INFO
> > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > > property: [key=policyLocation,
> > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > HTTP Server started at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/>
> >
> > >
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > Artemis Jolokia REST API available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > >
> > >
> > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > Console
> > > available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> > >
> > > 20:42:58,455 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl]
> > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d> > > > [ID=f85ec51b<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d>
> > [ID=f85ec51b>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616/>, remote=/127.0.0.6:49865<http://127.0.0.6:49865/>]]
> > > 2023-12-08 20:42:59,524 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858> > > > [ID=7cef23a2<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858>
> > [ID=7cef23a2>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616/>, remote=/127.0.0.6:59691<http://127.0.0.6:59691/>]]
> > > 2023-12-08 20:42:59,535 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 22, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,536 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,549 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 24, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,550 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,583 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: OK
> > > 2023-12-08 20:42:59,584 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@5f46053b
> > > 2023-12-08 20:42:59,593 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > 2023-12-08 20:42:59,594 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 28, result: OK
> > > 2023-12-08 20:42:59,616 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:59,615 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 31, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 32, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 35, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 39, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:43:08,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256> > > > [ID=1d1d79ce<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256>
> > [ID=1d1d79ce>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616/>, remote=/127.0.0.6:60787<http://127.0.0.6:60787/>]]
> > > 2023-12-08 20:43:08,464 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7> > > > [ID=8f88c6d5<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7>
> > [ID=8f88c6d5>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616/>, remote=/127.0.0.6:40065<http://127.0.0.6:40065/>]]
> > >
> > >
> > > HA “slave” broker log.
> > >
> > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > synchronized with live server,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > >
> > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221031: backup announced
> > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221084: Requested 0 quorum votes
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221071: Failing over based on quorum vote results.
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:51,946 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616/>, remote=/127.0.0.6:56167<http://127.0.0.6:56167/>]]
> > > 2023-12-08 20:42:56,977 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > destroyed
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616/>, remote=/127.0.0.6:56167<http://127.0.0.6:56167/>]]
> > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2023-12-08 20:42:57,445 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6
> > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > > Retrying
> > > 2023-12-08 20:42:57,629 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > > [ID=483170f6<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616/>, remote=/127.0.0.6:55911<http://127.0.0.6:55911/>]]
> > > 2023-12-08 20:42:57,630 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > > [ID=4514a2b3<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616/>, remote=/127.0.0.6:60199<http://127.0.0.6:60199/>]]
> > > 2023-12-08 20:42:57,648 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > > [ID=fcde4d78<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616/>, remote=/127.0.0.6:53221<http://127.0.0.6:53221/>]]
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 29, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 31, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,708 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 33, result: OK
> > > 2023-12-08 20:42:57,709 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@3645037f
> > > 2023-12-08 20:42:57,719 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 35, result: OK
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:57,753 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > 2023-12-08 20:42:57,760 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 38, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, > > > with transaction: null
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 41, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 43, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 44, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 46, result: OK
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 48, result: OK
> > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > >
> > >
> > >
> >
>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
RE: Trouble with Replication HA Master/Slave config performing failback
Posted by Lino Pereira <li...@redpointglobal.com.INVALID>.
Hi Paul and/or Justin,
How many backup instances are you using, or what is the minimum number you recommend?
Thanks,
Lino
[rg] <https://www.redpointglobal.com/>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: John Lilley <jo...@redpointglobal.com.INVALID>
Sent: Friday, January 19, 2024 9:55 AM
To: users@activemq.apache.org
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hi Paul,
Thanks for all of this! One thing you said struck me as odd: “You will need a separate PVC for both your Primary and standby broker instances.”
By “separate PVC” do you mean completely different volumes? Or a shared volume with a mount in each pod?
Does this mean that the primary and standby do not communicate anything via the lock file? Perhaps I am confused by the name “lock file” which implies some kind of concurrency management.
Instead of two separate PVC volumes, would it be OK to use a shared PVC (in Azure, this is an NFS-mounted FileShare volume) but set the primary and backup to use different directories within the volume? To save cost and complexity.
john
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,LKdBdA9ZlqkKcRHwqxRvdDjB6YSFFL85iRuneQ5IIX7R166K_wfNUyrSU_lkBlXtAMgOKLoJXoOr25uSqnFzJrqo96TIt3XByD_7y9-Dqr3X8jSw5CHJyztGzqjE&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Shields, Paul Michael <pa...@hpe.com>>
Sent: Friday, January 19, 2024 9:48 AM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hi John,
I am using the activemq-artemis-operator from ArtemisCloud.io<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fArtemisCloud.io&c=E,1,TV8NB-Ny1beQqMowEfKkewBSKybapQAV_KnDp6P5OgtMCv0-n88PFErhGYbfjzDTXTShqF8_nA7PcwZSbhhhq9T1SeZtmTI-yE7OpN_U2KevLKqACVtBmbc,&typo=1&ancr_add=1> That operator only creates the PVC when I specify persistenceEnabled=true. So that is how I create a PVC for each broker instance. I then have to customize the broker.xml file to only have the <node-manager-lock-directory> on the PVC and the othe Artemis data directories on the local pod filesystem.
If you are deploying directly to k8s without an operator then you could have persistenceEnabled=false, but would have to define a PVC that would mount the <node-manager-lock-directory> to the broker pod to store the server.lock file. You will need a separate PVC for both your Primary and standby broker instances.
Yes, I have attached the broker.xml file to this email. Note that the only difference between my primary and standby instance broker.xml files is the <ha-policy> definition. So, I will attach the whole broker.xml from my Primary and just post the standby <ha-policy> portion.
Show the PVC mount:
ncn-m001:~ # kubectl -n dvs exec -it cray-dvs-mqtt-ss-0 -- /bin/bash
[jboss@cray-dvs-mqtt-ss-0 ~]$ df
Filesystem 1K-blocks Used Available Use% Mounted on
containerd_overlayfs 676576488 219342084 457234404 33% /
tmpfs 65536 0 65536 0% /dev
tmpfs 131773112 0 131773112 0% /sys/fs/cgroup
containerd_overlayfs 676576488 219342084 457234404 33% /etc/hostname
shm 65536 0 65536 0% /dev/shm
/dev/sdc3 187445840 222380 187223460 1% /etc/hosts
/dev/rbd2 1992552 34876 1941292 2% /opt/cray-dvs-mqtt/data <--- this is the PVC mount
tmpfs 263443828 4 263443824 1% /amq/extra/secrets/cray-dvs-mqtt-props
tmpfs 263443828 12 263443816 1% /run/secrets///kubernetes.io/serviceaccount<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f%2f%2fkubernetes.io%2fserviceaccount&c=E,1,zx-JtrSyBrNbA01-NTG91tZA5WOC-kl6BPtfG3RTuIQykLDhNJnVTq3pNbooNd06f7PobYG4wYRoE_WMe607UGKQe_yDrIGm_N6BvubSj6O0VVM,&typo=1&ancr_add=1>
tmpfs 131773112 0 131773112 0% /proc/acpi
tmpfs 131773112 0 131773112 0% /proc/scsi
tmpfs 131773112 0 131773112 0% /sys/firmware
Show the lock file on the PVC
[jboss@cray-dvs-mqtt-ss-0 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/server.lock
0000000 3050 fa30 65aa b57a 117b 9eee b25c bf25
0000020 7183 0096
0000023
[jboss@cray-dvs-mqtt-ss-0 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/serverlock.1
0000000
[jboss@cray-dvs-mqtt-ss-0 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/serverlock.2
0000000
[jboss@cray-dvs-mqtt-ss-0 ~]$ exit
exit
ncn-m001:~ # kubectl -n dvs exec -it cray-dvs-mqtt-ss-1 -- /bin/bash
[jboss@cray-dvs-mqtt-ss-1 ~]$ ls -l /opt/cray-dvs-mqtt/data/journal/
total 4
-rw-r--r-- 1 jboss root 19 Jan 17 22:26 server.lock
-rw-r--r-- 1 jboss root 0 Jan 17 22:25 serverlock.1
-rw-r--r-- 1 jboss root 0 Jan 17 22:25 serverlock.2
[jboss@cray-dvs-mqtt-ss-1 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/server.lock
0000000 3030 fa30 65aa b57a 117b 9eee b25c bf25
0000020 7183 0096
0000023
[jboss@cray-dvs-mqtt-ss-1 ~]$
Primary instance broker.xml file: Note I have logging enable. You probably don’t want that in a production environment.
[jboss@cray-dvs-mqtt-ss-0 ~]$ cat amq-broker/etc/broker.xml
<?xml version='1.0'?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<configuration xmlns="urn:activemq"
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
xmlns:xi=http://www.w3.org/2001/XInclude
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
xsi:schemaLocation="urn:activemq:core ">
<name>amq-broker</name>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<persistence-enabled>true</persistence-enabled>
<!-- this could be ASYNCIO, MAPPED, NIO
ASYNCIO: Linux Libaio
MAPPED: mmap files
NIO: Plain Java Files
-->
<journal-type>NIO</journal-type>
<node-manager-lock-directory>/opt/cray-dvs-mqtt/data/journal</node-manager-lock-directory>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<!-- if you want to retain your journal uncomment this following configuration.
This will allow your system to keep 7 days of your data, up to 10G. Tweak it accordingly to your use case and capacity.
it is recommended to use a separate storage unit from the journal for performance considerations.
<journal-retention-directory period="7" unit="DAYS" storage-limit="10G">data/retention</journal-retention-directory>
You can also enable retention by using the argument journal-retention on the `artemis create` command -->
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<!--
You can verify the network health of a particular NIC by specifying the <network-check-NIC> element.
<network-check-NIC>theNicName</network-check-NIC>
-->
<!--
Use this to use an HTTP server to validate the network
<network-check-URL-list>http://www.apache.org</network-check-URL-list<http://www.apache.org%3c/network-check-URL-list>> -->
<!-- <network-check-period>10000</network-check-period> -->
<!-- <network-check-timeout>1000</network-check-timeout> -->
<!-- this is a comma separated list, no spaces, just DNS or IPs
it should accept IPV6
Warning: Make sure you understand your network topology as this is meant to validate if your network is valid.
Using IPs that could eventually disappear or be partially visible may defeat the purpose.
You can use a list of multiple IPs, and if any successful ping will make the server OK to continue running -->
<!-- <network-check-list>10.0.0.1</network-check-list> -->
<!-- use this to customize the ping used for ipv4 addresses -->
<!-- <network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command> -->
<!-- use this to customize the ping used for ipv6 addresses -->
<!-- <network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command> -->
<connectors>
<!-- Connector used to be announced through cluster connections and notifications -->
<connector name="artemis">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616</connector>
</connectors>
<!-- how often we are looking for how many bytes are being used on the disk in ms -->
<disk-scan-period>5000</disk-scan-period>
<!-- once the disk hits this limit the system will block, or close the connection in certain protocols
that won't support flow control. -->
<max-disk-usage>90</max-disk-usage>
<!-- should the broker detect dead locks and other issues -->
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<!-- the system will enter into page mode once you hit this limit. This is an estimate in bytes of how much the messages are using in memory
The system will use half of the available memory (-Xmx) by default for the global-max-size.
You may specify a different value here if you need to customize it to your needs.
<global-max-size>100Mb</global-max-size> -->
<!-- the maximum number of messages accepted before entering full address mode.
if global-max-size is specified the full address mode will be specified by whatever hits it first. -->
<global-max-messages>-1</global-max-messages>
<acceptors>
<!-- useEpoll means: it will use Netty epoll if you are on a system (Linux) that supports it -->
<!-- amqpCredits: The number of credits sent to AMQP producers -->
<!-- amqpLowCredits: The server will send the # credits specified at amqpCredits at this low mark -->
<!-- amqpDuplicateDetection: If you are not using duplicate detection, set this to false
as duplicate detection requires applicationProperties to be parsed on the server. -->
<!-- amqpMinLargeMessageSize: Determines how many bytes are considered large, so we start using files to hold their data.
default: 102400, -1 would mean to disable large mesasge control -->
<!-- Note: If an acceptor needs to be compatible with HornetQ and/or Artemis 1.x clients add
"anycastPrefix=jms.queue.;multicastPrefix=jms.topic." to the acceptor url.
See https://issues.apache.org/jira/browse/ARTEMIS-1644 for more information. -->
<!-- Acceptor for every supported protocol -->
<acceptor name="dvs">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883?protocols=MQTT;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor><acceptor name="scaleDown">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616?protocols=CORE;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor>
</acceptors>
<cluster-user>EtBU3LYU</cluster-user>
<cluster-password>xOKqDzwH</cluster-password>
<broadcast-groups>
<broadcast-group name="my-broadcast-group"> <jgroups-file>jgroups-ping.xml</jgroups-file> <jgroups-channel>activemq_broadcast_channel</jgroups-channel> <connector-ref>artemis</connector-ref> </broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group"> <jgroups-file>jgroups-ping.xml</jgroups-file> <jgroups-channel>activemq_broadcast_channel</jgroups-channel> <refresh-timeout>10000</refresh-timeout> </discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster"> <connector-ref>artemis</connector-ref> <retry-interval>1000</retry-interval> <retry-interval-multiplier>2</retry-interval-multiplier> <max-retry-interval>32000</max-retry-interval> <initial-connect-attempts>20</initial-connect-attempts> <reconnect-attempts>-1</reconnect-attempts> <use-duplicate-detection>true</use-duplicate-detection> <message-load-balancing>ON_DEMAND</message-load-balancing> <max-hops>1</max-hops> <discovery-group-ref discovery-group-name="my-discovery-group"/> </cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="admin"/>
<permission type="deleteNonDurableQueue" roles="admin"/>
<permission type="createDurableQueue" roles="admin"/>
<permission type="deleteDurableQueue" roles="admin"/>
<permission type="createAddress" roles="admin"/>
<permission type="deleteAddress" roles="admin"/>
<permission type="consume" roles="admin"/>
<permission type="browse" roles="admin"/>
<permission type="send" roles="admin"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="admin"/>
</security-setting>
</security-settings>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="//activemq.management#"><https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f%2f%2factivemq.management%23%26quot%3b%26gt%3b&c=E,1,BRroUY8kfUeX80gvrp7t72VVHVGMPASjpLV9yYkz4JVee8GpRfZvI_Kw2UOOUNV2uD0jK5gOPSY2kMhPWHhNzPMAHND9p06dXyXm9NI42fIRLB9HOfs,&typo=1&ancr_add=1>
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<redistribution-delay>0</redistribution-delay>
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- if max-size-bytes and max-size-messages were both enabled, the system will enter into paging
based on the first attribute to hits the maximum value -->
<!-- limit for the address in bytes, -1 means unlimited -->
<max-size-bytes>-1</max-size-bytes>
<!-- limit for the address in messages, -1 means unlimited -->
<max-size-messages>-1</max-size-messages>
<!-- the size of each file on paging. Notice we keep files in memory while they are in use.
Lower this setting if you have too many queues in memory. -->
<page-size-bytes>10M</page-size-bytes>
<!-- limit how many messages are read from paging into the Queue. -->
<max-read-page-messages>-1</max-read-page-messages>
<!-- limit how much memory is read from paging into the Queue. -->
<max-read-page-bytes>20M</max-read-page-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-delete-queues>false</auto-delete-queues>
<auto-delete-addresses>false</auto-delete-addresses>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
<broker-plugins>
<broker-plugin class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
<property key="LOG_ALL_EVENTS" value="true"/>
<property key="LOG_CONNECTION_EVENTS" value="true"/>
<property key="LOG_SESSION_EVENTS" value="true"/>
<property key="LOG_CONSUMER_EVENTS" value="true"/>
<property key="LOG_DELIVERING_EVENTS" value="true"/>
<property key="LOG_SENDING_EVENTS" value="true"/>
<property key="LOG_INTERNAL_EVENTS" value="true"/>
</broker-plugin>
</broker-plugins>
<metrics> <plugin class-name="com.redhat.amq.broker.core.server.metrics.plugins.ArtemisPrometheusMetricsPlugin"/> </metrics>
</core>
</configuration>
[jboss@cray-dvs-mqtt-ss-0 ~]$
Standby instance broker.xml <ha-policy> stanza
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Date: Friday, January 19, 2024 at 10:21 AM
To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
Cc: Lino Pereira <li...@redpointglobal.com>>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Paul,
Thanks for the information! It sounds like you have experience with AMQ in K8S. Can you share your two broker.xml files with us?
Lino: do we currently set persistenceEnable=false? I would expect there’s no reason for setting it true, since the pod-local storage will evaporate with the pod.
Thanks
John
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,_MEdc5Qy-vTEnjozH--yv7M1N-lLpEcvjRVTe0TZfe8G_dcxUJzpNCwemRvf2WA7KwCsV-QQ_ejGemmADMP_e4L0WvHHBpjTniOeKgAhT3d5l10,&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Shields, Paul Michael <pa...@hpe.com>>
Sent: Friday, January 19, 2024 8:51 AM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hi John,
The <node-manager-lock-directory>. Needs to be specified in both the master and slave broker.xml files.
the nodeUUID which is contained in the server lock file was changing between active broker startups. The server.lock file is stored in the /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our current configuration of the broker has persistenceEnable=false which causes any data/broker state to be removed when the broker pod is destroyed. Thus a new nodeUUID was being generated on each Primary broker startup. When the Primary broker instance starts up it looks for an existing server.lock file. When it does not find one it generates a new nodeUUID and places it in the server.lock file, otherwise it uses the nodeUUID contained within the existing server.lock file. The Primary broker then communicates this nodeUUID through the cluster connection to the Standby broker instance on startup. The Standby broker instance uses the nodeUUID to determine if the Primary broker instance is part of its HA pair. The Standby broker instance saved off the original nodeUUID when the HA cluster was first started and since the recently communicated nodeUUID does not match the nodeUUID that it previously stored the fail back is not performed. Once persistence was enabled the Standby broker instance performed the fallback to the Primary instance and was no longer in the "active" state
This is what I have observed in our Kubernetes environment. Justin please make any corrections to my description of the nodeUUID use.
Paul
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Date: Thursday, January 18, 2024 at 5:49 PM
To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Justin,
Can you elaborate on the lock file in the <node-manager-lock-directory>?
Does it need to be specified in the broker.xml for both the master and the slave?
Does it actually get used as a “lock” in the concurrency sense, to coordinate the live/backup election?
Thanks
john
[rg]<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,0rYuECbqGwpo9IjFKJrQQrfyzL5V_aEiqSXV25I7DeF4wuQn3I6z1sJCY1NkNJ-bjdpV8nVix_VDU0mK5tgrxuSQaSyrlVLxE5EiTf0Z8G4F&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Monday, January 15, 2024 9:56 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
In the replication case you still need a PV in K8s, but I don't believe you need to put the whole journal on the PV (and suffer the performance penalty). You should just need to point the node-manager-lock-directory in broker.xml to the PV.
Justin
On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
Based on your reply in the provided link, are you saying that even for the replication HA case, you still need an external FileShare PV, in K8s, so that the nodeID can be persisted between restarts. In this use case, is the performance of this FileShare less of a concern than it is for the shared-store use case?
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
-----Original Message-----
From: Justin Bertram <jb...@apache.org>>
Sent: Friday, January 12, 2024 4:53 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
I sent that email on December 15. You can find it on the web-based mailing
list interface [1].
Justin
[1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l
On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <pa...@hpe.com>>
wrote:
> Hi Justin,
>
> Forgive me but I don’t remember a journal persistence issue discussion. I
> did an email search too and did not see that discussion either. Would you
> please refresh me on the discussion? I may have inadvertently deleted that?
>
> I do see data being replicated to my production slave/standby instance on
> initial startup of the two broker instances. Failover works as expected.
> But when the primary instance starts backup I do not see the active slave
> replicate to the starting primary and I do not see the active slave restart
> and go to the backup server state.
>
> What is the communication sequence when the primary starts up after it
> quit with the active backup broker? In other words, how is the active
> backup broker notified to start replication to the newly started primary
> broker?
>
> I ran the replicated-failback example again with TRACE level logging
> turned on and I do see the attemptFailback=false message in the
> target/server1/log/artemis.log file at startup, so it must be changed later
> or am I not understanding the purpose of this flag?
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> discoveryInitialWaitTimeout=10000}]
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost, b=null],
> nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost)
>
> From my production slave broker log
>
>
> 024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> Thanks,
> Paul
> From: Justin Bertram <jb...@apache.org>>
> Date: Friday, January 12, 2024 at 3:38 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > Is it possible to use jgroups instead of UDP for the cluster connection
> in the example?
>
> Yes. Check out examples/features/clustered/clustered-jgroups for an example
> configuration. Keep in mind that UDP, JGroups, etc. is only used for
> initial discovery. Once brokers are discovered then normal TCP connections
> are created.
>
> Before I continue looking through your logs, etc. can you confirm that you
> resolved the journal persistence issue I described in my previous email? If
> that's not resolved there's no chance for failback to occur.
>
>
> Justin
>
>
>
> On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>>
> wrote:
>
> > Hi Justin,
> >
> > I tried the example and it works. I turned up logging to get a success
> > path through the code to maybe see where things get de-railed. Is it
> > possible to use jgroups instead of UDP for the cluster connection in the
> > example? That would give me a better success signature.
> >
> > My standby instance has this in the broker.xml which is the same as in
> the
> > example:
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > I am seeing this series of log messages that suggests that failback is
> not
> > enabled. Is the attemptFailback=false the default at this point in
> startup
> > and it is changed later or it there something gone wrong at this point?
> >
> > 2024-01-11 18:24:14,066 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > QuorumManager(server=null) about node up =
> > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> > <mailto:
<mailto:%0b>> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> >
> > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > ClientSessionFactoryImpl received backup update for live/backup pair =
> > TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > / null but it didn't belong to TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,067 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Cluster Connected
> >
> > 2024-01-11 18:24:14,068 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting backup manager
> >
> > 2024-01-11 18:24:14,069 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > backup config ClusterConnectionConfiguration{name='my-cluster',
> address='',
> > connectorName='artemis', clientFailureCheckPeriod=30000,
> > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > clusterNotificationAttempts=2}
> >
> > 2024-01-11 18:24:14,081 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > 2024-01-11 18:24:14,083 TRACE
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> >
> > java.lang.Exception:trace<ftp://java.lang.Exception/trace>
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4 rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying reconnection attempt 0/0
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18> > > <mailto:connectorFactory<mailto:connectorFactory>
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18>
> >
> > and currentConnectorConfig: TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Connector NettyConnector
> > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using native epoll
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Started EPOLL Netty Connector version 4.1.86.Final to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Remote destination:
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.25:61616<ftp://10.44.0.25/61616>
> >
> >
> > Thanks,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>>
> > Date: Wednesday, December 13, 2023 at 3:23 PM
> > To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > ActiveMQ Artemis ships with an example in the
> > examples/features/ha/replicated-failback directory that demonstrates how
> > this works. I ran this example and it worked fine. I also started up the
> > brokers manually and triggered failover and failback. Again, everything
> > worked fine. My only guess is that there's something configured wrong or
> > perhaps there's an environmental issue that's preventing failback from
> > working properly for you. If you could provide clear steps on how to
> > reproduce this issue I could investigate further.
> >
> >
> > Justin
> >
> > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am having an issue with failback to the master broker. I start the
> > > master broker then the slave broker. I see the replication from the
> > master
> > > to the slave happen. I stop the master/active broker and a failover
> > happens
> > > to the slave as expected. When the master is started a failback to it
> is
> > > expected but does not happen. The following is my ha-policy config.
> > Did I
> > > miss something in the ha-policy configuration? We are using version
> > 2.28.0
> > > and java17.
> > >
> > > Thanks,
> > > Paul
> > >
> > >
> > > Master broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <master>
> > >
> > > <check-for-live-server>true</check-for-live-server>
> > >
> > > </master>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > Slave broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > The following are the logs from the master and slave brokers
> > > HA “master” broker log after restart
> > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > Starting embedded web server
> > > 2023-12-08 20:42:57,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 15, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,627 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3
> > > 2023-12-08 20:42:57,657 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2> > > > [ID=47edf16f<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2>
> > [ID=47edf16f>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/10.44.0.18:34508<http://10.44.0.18:34508>]]
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > > activemq-branding plugin
> > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > artemis-plugin plugin
> > > 2023-12-08 20:42:57,766 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 18, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,767 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 20, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > Initialising
> > > hawtio services
> > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > Configuration
> > > will be discovered via system properties
> > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > Hawtio 2.15.0
> > > 2023-12-08 20:42:57,975 INFO
> > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > authentication filter, JAAS realm: "activemq" authorized role(s):
> "admin"
> > > role principal classes:
> > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > servlet is disabled
> > > 2023-12-08 20:42:57,989 INFO
> > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > > property: [key=policyLocation,
> > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > HTTP Server started at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > >
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > Artemis Jolokia REST API available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > >
> > >
> > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > Console
> > > available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> > >
> > > 20:42:58,455 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl]
> > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d> > > > [ID=f85ec51b<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d>
> > [ID=f85ec51b>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:49865<http://127.0.0.6:49865>]]
> > > 2023-12-08 20:42:59,524 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858> > > > [ID=7cef23a2<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858>
> > [ID=7cef23a2>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:59691<http://127.0.0.6:59691>]]
> > > 2023-12-08 20:42:59,535 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 22, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,536 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,549 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 24, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,550 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,583 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: OK
> > > 2023-12-08 20:42:59,584 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@5f46053b
> > > 2023-12-08 20:42:59,593 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > 2023-12-08 20:42:59,594 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 28, result: OK
> > > 2023-12-08 20:42:59,616 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:59,615 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 31, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 32, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 35, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 39, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:43:08,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256> > > > [ID=1d1d79ce<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256>
> > [ID=1d1d79ce>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:60787<http://127.0.0.6:60787>]]
> > > 2023-12-08 20:43:08,464 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7> > > > [ID=8f88c6d5<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7>
> > [ID=8f88c6d5>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:40065<http://127.0.0.6:40065>]]
> > >
> > >
> > > HA “slave” broker log.
> > >
> > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > synchronized with live server,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > >
> > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221031: backup announced
> > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221084: Requested 0 quorum votes
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221071: Failing over based on quorum vote results.
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:51,946 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:56,977 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > destroyed
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2023-12-08 20:42:57,445 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6
> > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > > Retrying
> > > 2023-12-08 20:42:57,629 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > > [ID=483170f6<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:55911<http://127.0.0.6:55911>]]
> > > 2023-12-08 20:42:57,630 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > > [ID=4514a2b3<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:60199<http://127.0.0.6:60199>]]
> > > 2023-12-08 20:42:57,648 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > > [ID=fcde4d78<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:53221<http://127.0.0.6:53221>]]
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 29, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 31, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,708 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 33, result: OK
> > > 2023-12-08 20:42:57,709 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@3645037f
> > > 2023-12-08 20:42:57,719 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 35, result: OK
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:57,753 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > 2023-12-08 20:42:57,760 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 38, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, > > > with transaction: null
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 41, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 43, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 44, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 46, result: OK
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 48, result: OK
> > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > >
> > >
> > >
> >
>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by "Shields, Paul Michael" <pa...@hpe.com>.
Hi John,
I use two different volumes, but that is because of the operator that I am using.
The server.lock files contain nodeUUID. The nodeUUIDs are used for HA pairing, if they don’t match a HA pair is not formed.
I suppose you could place two different directories on a single PVC to save cost. But you would not want to share the <node-manager-lock-directory> between the two broker instances. You could specify different paths on the same PVC. For the primary/master broker.xml file have an entry like <node-manager-lock-directory>/$MY-PVC/server0</node-manager-lock-directory>. And on the standby/slave instance broker.xml file have an entry like <node-manager-lock-directory>/$MY-PVC/server1</node-manager-lock-directory>.
Paul
From: John Lilley <jo...@redpointglobal.com.INVALID>
Date: Friday, January 19, 2024 at 10:56 AM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Paul,
Thanks for all of this! One thing you said struck me as odd: “You will need a separate PVC for both your Primary and standby broker instances.”
By “separate PVC” do you mean completely different volumes? Or a shared volume with a mount in each pod?
Does this mean that the primary and standby do not communicate anything via the lock file? Perhaps I am confused by the name “lock file” which implies some kind of concurrency management.
Instead of two separate PVC volumes, would it be OK to use a shared PVC (in Azure, this is an NFS-mounted FileShare volume) but set the primary and backup to use different directories within the volume? To save cost and complexity.
john
[rg]<https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Shields, Paul Michael <pa...@hpe.com>
Sent: Friday, January 19, 2024 9:48 AM
To: users@activemq.apache.org
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hi John,
I am using the activemq-artemis-operator from ArtemisCloud.io<https://linkprotect.cudasvc.com/url?a=https://ArtemisCloud.io&c=E,1,TV8NB-Ny1beQqMowEfKkewBSKybapQAV_KnDp6P5OgtMCv0-n88PFErhGYbfjzDTXTShqF8_nA7PcwZSbhhhq9T1SeZtmTI-yE7OpN_U2KevLKqACVtBmbc,&typo=1&ancr_add=1> That operator only creates the PVC when I specify persistenceEnabled=true. So that is how I create a PVC for each broker instance. I then have to customize the broker.xml file to only have the <node-manager-lock-directory> on the PVC and the othe Artemis data directories on the local pod filesystem.
If you are deploying directly to k8s without an operator then you could have persistenceEnabled=false, but would have to define a PVC that would mount the <node-manager-lock-directory> to the broker pod to store the server.lock file. You will need a separate PVC for both your Primary and standby broker instances.
Yes, I have attached the broker.xml file to this email. Note that the only difference between my primary and standby instance broker.xml files is the <ha-policy> definition. So, I will attach the whole broker.xml from my Primary and just post the standby <ha-policy> portion.
Show the PVC mount:
ncn-m001:~ # kubectl -n dvs exec -it cray-dvs-mqtt-ss-0 -- /bin/bash
[jboss@cray-dvs-mqtt-ss-0 ~]$ df
Filesystem 1K-blocks Used Available Use% Mounted on
containerd_overlayfs 676576488 219342084 457234404 33% /
tmpfs 65536 0 65536 0% /dev
tmpfs 131773112 0 131773112 0% /sys/fs/cgroup
containerd_overlayfs 676576488 219342084 457234404 33% /etc/hostname
shm 65536 0 65536 0% /dev/shm
/dev/sdc3 187445840 222380 187223460 1% /etc/hosts
/dev/rbd2 1992552 34876 1941292 2% /opt/cray-dvs-mqtt/data <--- this is the PVC mount
tmpfs 263443828 4 263443824 1% /amq/extra/secrets/cray-dvs-mqtt-props
tmpfs 263443828 12 263443816 1% /run/secrets///kubernetes.io/serviceaccount<https://linkprotect.cudasvc.com/url?a=https:////kubernetes.io/serviceaccount&c=E,1,zx-JtrSyBrNbA01-NTG91tZA5WOC-kl6BPtfG3RTuIQykLDhNJnVTq3pNbooNd06f7PobYG4wYRoE_WMe607UGKQe_yDrIGm_N6BvubSj6O0VVM,&typo=1&ancr_add=1>
tmpfs 131773112 0 131773112 0% /proc/acpi
tmpfs 131773112 0 131773112 0% /proc/scsi
tmpfs 131773112 0 131773112 0% /sys/firmware
Show the lock file on the PVC
[jboss@cray-dvs-mqtt-ss-0 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/server.lock
0000000 3050 fa30 65aa b57a 117b 9eee b25c bf25
0000020 7183 0096
0000023
[jboss@cray-dvs-mqtt-ss-0 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/serverlock.1
0000000
[jboss@cray-dvs-mqtt-ss-0 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/serverlock.2
0000000
[jboss@cray-dvs-mqtt-ss-0 ~]$ exit
exit
ncn-m001:~ # kubectl -n dvs exec -it cray-dvs-mqtt-ss-1 -- /bin/bash
[jboss@cray-dvs-mqtt-ss-1 ~]$ ls -l /opt/cray-dvs-mqtt/data/journal/
total 4
-rw-r--r-- 1 jboss root 19 Jan 17 22:26 server.lock
-rw-r--r-- 1 jboss root 0 Jan 17 22:25 serverlock.1
-rw-r--r-- 1 jboss root 0 Jan 17 22:25 serverlock.2
[jboss@cray-dvs-mqtt-ss-1 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/server.lock
0000000 3030 fa30 65aa b57a 117b 9eee b25c bf25
0000020 7183 0096
0000023
[jboss@cray-dvs-mqtt-ss-1 ~]$
Primary instance broker.xml file: Note I have logging enable. You probably don’t want that in a production environment.
[jboss@cray-dvs-mqtt-ss-0 ~]$ cat amq-broker/etc/broker.xml
<?xml version='1.0'?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0<http://www.apache.org/licenses/LICENSE-2.0>
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<configuration xmlns="urn:activemq"
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance<http://www.w3.org/2001/XMLSchema-instance>
xmlns:xi=http://www.w3.org/2001/XInclude<http://www.w3.org/2001/XInclude>
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance<http://www.w3.org/2001/XMLSchema-instance>
xsi:schemaLocation="urn:activemq:core ">
<name>amq-broker</name>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<persistence-enabled>true</persistence-enabled>
<!-- this could be ASYNCIO, MAPPED, NIO
ASYNCIO: Linux Libaio
MAPPED: mmap files
NIO: Plain Java Files
-->
<journal-type>NIO</journal-type>
<node-manager-lock-directory>/opt/cray-dvs-mqtt/data/journal</node-manager-lock-directory>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<!-- if you want to retain your journal uncomment this following configuration.
This will allow your system to keep 7 days of your data, up to 10G. Tweak it accordingly to your use case and capacity.
it is recommended to use a separate storage unit from the journal for performance considerations.
<journal-retention-directory period="7" unit="DAYS" storage-limit="10G">data/retention</journal-retention-directory>
You can also enable retention by using the argument journal-retention on the `artemis create` command -->
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<!--
You can verify the network health of a particular NIC by specifying the <network-check-NIC> element.
<network-check-NIC>theNicName</network-check-NIC>
-->
<!--
Use this to use an HTTP server to validate the network
<network-check-URL-list>http://www.apache.org</network-check-URL-list<http://www.apache.org</network-check-URL-list>> -->
<!-- <network-check-period>10000</network-check-period> -->
<!-- <network-check-timeout>1000</network-check-timeout> -->
<!-- this is a comma separated list, no spaces, just DNS or IPs
it should accept IPV6
Warning: Make sure you understand your network topology as this is meant to validate if your network is valid.
Using IPs that could eventually disappear or be partially visible may defeat the purpose.
You can use a list of multiple IPs, and if any successful ping will make the server OK to continue running -->
<!-- <network-check-list>10.0.0.1</network-check-list> -->
<!-- use this to customize the ping used for ipv4 addresses -->
<!-- <network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command> -->
<!-- use this to customize the ping used for ipv6 addresses -->
<!-- <network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command> -->
<connectors>
<!-- Connector used to be announced through cluster connections and notifications -->
<connector name="artemis">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616</connector>
</connectors>
<!-- how often we are looking for how many bytes are being used on the disk in ms -->
<disk-scan-period>5000</disk-scan-period>
<!-- once the disk hits this limit the system will block, or close the connection in certain protocols
that won't support flow control. -->
<max-disk-usage>90</max-disk-usage>
<!-- should the broker detect dead locks and other issues -->
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<!-- the system will enter into page mode once you hit this limit. This is an estimate in bytes of how much the messages are using in memory
The system will use half of the available memory (-Xmx) by default for the global-max-size.
You may specify a different value here if you need to customize it to your needs.
<global-max-size>100Mb</global-max-size> -->
<!-- the maximum number of messages accepted before entering full address mode.
if global-max-size is specified the full address mode will be specified by whatever hits it first. -->
<global-max-messages>-1</global-max-messages>
<acceptors>
<!-- useEpoll means: it will use Netty epoll if you are on a system (Linux) that supports it -->
<!-- amqpCredits: The number of credits sent to AMQP producers -->
<!-- amqpLowCredits: The server will send the # credits specified at amqpCredits at this low mark -->
<!-- amqpDuplicateDetection: If you are not using duplicate detection, set this to false
as duplicate detection requires applicationProperties to be parsed on the server. -->
<!-- amqpMinLargeMessageSize: Determines how many bytes are considered large, so we start using files to hold their data.
default: 102400, -1 would mean to disable large mesasge control -->
<!-- Note: If an acceptor needs to be compatible with HornetQ and/or Artemis 1.x clients add
"anycastPrefix=jms.queue.;multicastPrefix=jms.topic." to the acceptor url.
See https://issues.apache.org/jira/browse/ARTEMIS-1644<https://issues.apache.org/jira/browse/ARTEMIS-1644> for more information. -->
<!-- Acceptor for every supported protocol -->
<acceptor name="dvs">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883?protocols=MQTT;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor><acceptor name="scaleDown">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616?protocols=CORE;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor>
</acceptors>
<cluster-user>EtBU3LYU</cluster-user>
<cluster-password>xOKqDzwH</cluster-password>
<broadcast-groups>
<broadcast-group name="my-broadcast-group"> <jgroups-file>jgroups-ping.xml</jgroups-file> <jgroups-channel>activemq_broadcast_channel</jgroups-channel> <connector-ref>artemis</connector-ref> </broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group"> <jgroups-file>jgroups-ping.xml</jgroups-file> <jgroups-channel>activemq_broadcast_channel</jgroups-channel> <refresh-timeout>10000</refresh-timeout> </discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster"> <connector-ref>artemis</connector-ref> <retry-interval>1000</retry-interval> <retry-interval-multiplier>2</retry-interval-multiplier> <max-retry-interval>32000</max-retry-interval> <initial-connect-attempts>20</initial-connect-attempts> <reconnect-attempts>-1</reconnect-attempts> <use-duplicate-detection>true</use-duplicate-detection> <message-load-balancing>ON_DEMAND</message-load-balancing> <max-hops>1</max-hops> <discovery-group-ref discovery-group-name="my-discovery-group"/> </cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="admin"/>
<permission type="deleteNonDurableQueue" roles="admin"/>
<permission type="createDurableQueue" roles="admin"/>
<permission type="deleteDurableQueue" roles="admin"/>
<permission type="createAddress" roles="admin"/>
<permission type="deleteAddress" roles="admin"/>
<permission type="consume" roles="admin"/>
<permission type="browse" roles="admin"/>
<permission type="send" roles="admin"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="admin"/>
</security-setting>
</security-settings>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="//activemq.management#"><https://linkprotect.cudasvc.com/url?a=https:////activemq.management#">&c=E,1,BRroUY8kfUeX80gvrp7t72VVHVGMPASjpLV9yYkz4JVee8GpRfZvI_Kw2UOOUNV2uD0jK5gOPSY2kMhPWHhNzPMAHND9p06dXyXm9NI42fIRLB9HOfs,&typo=1&ancr_add=1>
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<redistribution-delay>0</redistribution-delay>
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- if max-size-bytes and max-size-messages were both enabled, the system will enter into paging
based on the first attribute to hits the maximum value -->
<!-- limit for the address in bytes, -1 means unlimited -->
<max-size-bytes>-1</max-size-bytes>
<!-- limit for the address in messages, -1 means unlimited -->
<max-size-messages>-1</max-size-messages>
<!-- the size of each file on paging. Notice we keep files in memory while they are in use.
Lower this setting if you have too many queues in memory. -->
<page-size-bytes>10M</page-size-bytes>
<!-- limit how many messages are read from paging into the Queue. -->
<max-read-page-messages>-1</max-read-page-messages>
<!-- limit how much memory is read from paging into the Queue. -->
<max-read-page-bytes>20M</max-read-page-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-delete-queues>false</auto-delete-queues>
<auto-delete-addresses>false</auto-delete-addresses>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
<broker-plugins>
<broker-plugin class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
<property key="LOG_ALL_EVENTS" value="true"/>
<property key="LOG_CONNECTION_EVENTS" value="true"/>
<property key="LOG_SESSION_EVENTS" value="true"/>
<property key="LOG_CONSUMER_EVENTS" value="true"/>
<property key="LOG_DELIVERING_EVENTS" value="true"/>
<property key="LOG_SENDING_EVENTS" value="true"/>
<property key="LOG_INTERNAL_EVENTS" value="true"/>
</broker-plugin>
</broker-plugins>
<metrics> <plugin class-name="com.redhat.amq.broker.core.server.metrics.plugins.ArtemisPrometheusMetricsPlugin"/> </metrics>
</core>
</configuration>
[jboss@cray-dvs-mqtt-ss-0 ~]$
Standby instance broker.xml <ha-policy> stanza
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Date: Friday, January 19, 2024 at 10:21 AM
To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
Cc: Lino Pereira <li...@redpointglobal.com>>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Paul,
Thanks for the information! It sounds like you have experience with AMQ in K8S. Can you share your two broker.xml files with us?
Lino: do we currently set persistenceEnable=false? I would expect there’s no reason for setting it true, since the pod-local storage will evaporate with the pod.
Thanks
John
[rg]<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,_MEdc5Qy-vTEnjozH--yv7M1N-lLpEcvjRVTe0TZfe8G_dcxUJzpNCwemRvf2WA7KwCsV-QQ_ejGemmADMP_e4L0WvHHBpjTniOeKgAhT3d5l10,&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Shields, Paul Michael <pa...@hpe.com>>
Sent: Friday, January 19, 2024 8:51 AM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hi John,
The <node-manager-lock-directory>. Needs to be specified in both the master and slave broker.xml files.
the nodeUUID which is contained in the server lock file was changing between active broker startups. The server.lock file is stored in the /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our current configuration of the broker has persistenceEnable=false which causes any data/broker state to be removed when the broker pod is destroyed. Thus a new nodeUUID was being generated on each Primary broker startup. When the Primary broker instance starts up it looks for an existing server.lock file. When it does not find one it generates a new nodeUUID and places it in the server.lock file, otherwise it uses the nodeUUID contained within the existing server.lock file. The Primary broker then communicates this nodeUUID through the cluster connection to the Standby broker instance on startup. The Standby broker instance uses the nodeUUID to determine if the Primary broker instance is part of its HA pair. The Standby broker instance saved off the original nodeUUID when the HA cluster was first started and since the recently communicated nodeUUID does not match the nodeUUID that it previously stored the fail back is not performed. Once persistence was enabled the Standby broker instance performed the fallback to the Primary instance and was no longer in the "active" state
This is what I have observed in our Kubernetes environment. Justin please make any corrections to my description of the nodeUUID use.
Paul
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Date: Thursday, January 18, 2024 at 5:49 PM
To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Justin,
Can you elaborate on the lock file in the <node-manager-lock-directory>?
Does it need to be specified in the broker.xml for both the master and the slave?
Does it actually get used as a “lock” in the concurrency sense, to coordinate the live/backup election?
Thanks
john
[rg]<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,0rYuECbqGwpo9IjFKJrQQrfyzL5V_aEiqSXV25I7DeF4wuQn3I6z1sJCY1NkNJ-bjdpV8nVix_VDU0mK5tgrxuSQaSyrlVLxE5EiTf0Z8G4F&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Monday, January 15, 2024 9:56 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
In the replication case you still need a PV in K8s, but I don't believe you need to put the whole journal on the PV (and suffer the performance penalty). You should just need to point the node-manager-lock-directory in broker.xml to the PV.
Justin
On Mon, Jan 15, 2024 at 4:35?PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
Based on your reply in the provided link, are you saying that even for the replication HA case, you still need an external FileShare PV, in K8s, so that the nodeID can be persisted between restarts. In this use case, is the performance of this FileShare less of a concern than it is for the shared-store use case?
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
-----Original Message-----
From: Justin Bertram <jb...@apache.org>>
Sent: Friday, January 12, 2024 4:53 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
I sent that email on December 15. You can find it on the web-based mailing
list interface [1].
Justin
[1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l<https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l>
On Fri, Jan 12, 2024 at 4:45?PM Shields, Paul Michael <pa...@hpe.com>>
wrote:
> Hi Justin,
>
> Forgive me but I don’t remember a journal persistence issue discussion. I
> did an email search too and did not see that discussion either. Would you
> please refresh me on the discussion? I may have inadvertently deleted that?
>
> I do see data being replicated to my production slave/standby instance on
> initial startup of the two broker instances. Failover works as expected.
> But when the primary instance starts backup I do not see the active slave
> replicate to the starting primary and I do not see the active slave restart
> and go to the backup server state.
>
> What is the communication sequence when the primary starts up after it
> quit with the active backup broker? In other words, how is the active
> backup broker notified to start replication to the newly started primary
> broker?
>
> I ran the replicated-failback example again with TRACE level logging
> turned on and I do see the attemptFailback=false message in the
> target/server1/log/artemis.log file at startup, so it must be changed later
> or am I not understanding the purpose of this flag?
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> discoveryInitialWaitTimeout=10000}]
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost, b=null],
> nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost)
>
> From my production slave broker log
>
>
> 024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> Thanks,
> Paul
> From: Justin Bertram <jb...@apache.org>>
> Date: Friday, January 12, 2024 at 3:38 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > Is it possible to use jgroups instead of UDP for the cluster connection
> in the example?
>
> Yes. Check out examples/features/clustered/clustered-jgroups for an example
> configuration. Keep in mind that UDP, JGroups, etc. is only used for
> initial discovery. Once brokers are discovered then normal TCP connections
> are created.
>
> Before I continue looking through your logs, etc. can you confirm that you
> resolved the journal persistence issue I described in my previous email? If
> that's not resolved there's no chance for failback to occur.
>
>
> Justin
>
>
>
> On Thu, Jan 11, 2024 at 3:09?PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>>
> wrote:
>
> > Hi Justin,
> >
> > I tried the example and it works. I turned up logging to get a success
> > path through the code to maybe see where things get de-railed. Is it
> > possible to use jgroups instead of UDP for the cluster connection in the
> > example? That would give me a better success signature.
> >
> > My standby instance has this in the broker.xml which is the same as in
> the
> > example:
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > I am seeing this series of log messages that suggests that failback is
> not
> > enabled. Is the attemptFailback=false the default at this point in
> startup
> > and it is changed later or it there something gone wrong at this point?
> >
> > 2024-01-11 18:24:14,066 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > QuorumManager(server=null) about node up =
> > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> > <mailto:
<mailto:%0b>> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> >
> > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > ClientSessionFactoryImpl received backup update for live/backup pair =
> > TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > / null but it didn't belong to TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,067 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Cluster Connected
> >
> > 2024-01-11 18:24:14,068 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting backup manager
> >
> > 2024-01-11 18:24:14,069 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > backup config ClusterConnectionConfiguration{name='my-cluster',
> address='',
> > connectorName='artemis', clientFailureCheckPeriod=30000,
> > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > clusterNotificationAttempts=2}
> >
> > 2024-01-11 18:24:14,081 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > 2024-01-11 18:24:14,083 TRACE
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> >
> > java.lang.Exception:trace<ftp://java.lang.Exception/trace>
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4 rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying reconnection attempt 0/0
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18> > > <mailto:connectorFactory<mailto:connectorFactory>
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18>
> >
> > and currentConnectorConfig: TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Connector NettyConnector
> > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using native epoll
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Started EPOLL Netty Connector version 4.1.86.Final to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Remote destination:
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.25:61616<ftp://10.44.0.25/61616>
> >
> >
> > Thanks,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>>
> > Date: Wednesday, December 13, 2023 at 3:23 PM
> > To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > ActiveMQ Artemis ships with an example in the
> > examples/features/ha/replicated-failback directory that demonstrates how
> > this works. I ran this example and it worked fine. I also started up the
> > brokers manually and triggered failover and failback. Again, everything
> > worked fine. My only guess is that there's something configured wrong or
> > perhaps there's an environmental issue that's preventing failback from
> > working properly for you. If you could provide clear steps on how to
> > reproduce this issue I could investigate further.
> >
> >
> > Justin
> >
> > On Fri, Dec 8, 2023 at 3:45?PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am having an issue with failback to the master broker. I start the
> > > master broker then the slave broker. I see the replication from the
> > master
> > > to the slave happen. I stop the master/active broker and a failover
> > happens
> > > to the slave as expected. When the master is started a failback to it
> is
> > > expected but does not happen. The following is my ha-policy config.
> > Did I
> > > miss something in the ha-policy configuration? We are using version
> > 2.28.0
> > > and java17.
> > >
> > > Thanks,
> > > Paul
> > >
> > >
> > > Master broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <master>
> > >
> > > <check-for-live-server>true</check-for-live-server>
> > >
> > > </master>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > Slave broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > The following are the logs from the master and slave brokers
> > > HA “master” broker log after restart
> > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > Starting embedded web server
> > > 2023-12-08 20:42:57,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 15, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,627 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3
> > > 2023-12-08 20:42:57,657 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2> > > > [ID=47edf16f<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2>
> > [ID=47edf16f>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/10.44.0.18:34508<http://10.44.0.18:34508>]]
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > > activemq-branding plugin
> > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > artemis-plugin plugin
> > > 2023-12-08 20:42:57,766 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 18, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,767 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 20, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > Initialising
> > > hawtio services
> > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > Configuration
> > > will be discovered via system properties
> > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > Hawtio 2.15.0
> > > 2023-12-08 20:42:57,975 INFO
> > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > authentication filter, JAAS realm: "activemq" authorized role(s):
> "admin"
> > > role principal classes:
> > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > servlet is disabled
> > > 2023-12-08 20:42:57,989 INFO
> > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > > property: [key=policyLocation,
> > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > HTTP Server started at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> >
> > >
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > Artemis Jolokia REST API available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08> > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08> > >
> > >
> > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > Console
> > > available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08>
> >
> > >
> > > 20:42:58,455 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl]
> > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d> > > > [ID=f85ec51b<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d>
> > [ID=f85ec51b>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:49865<http://127.0.0.6:49865>]]
> > > 2023-12-08 20:42:59,524 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858> > > > [ID=7cef23a2<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858>
> > [ID=7cef23a2>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:59691<http://127.0.0.6:59691>]]
> > > 2023-12-08 20:42:59,535 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 22, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,536 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,549 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 24, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,550 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,583 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: OK
> > > 2023-12-08 20:42:59,584 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@5f46053b
> > > 2023-12-08 20:42:59,593 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > 2023-12-08 20:42:59,594 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 28, result: OK
> > > 2023-12-08 20:42:59,616 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:59,615 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 31, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 32, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 35, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 39, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:43:08,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256> > > > [ID=1d1d79ce<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256>
> > [ID=1d1d79ce>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:60787<http://127.0.0.6:60787>]]
> > > 2023-12-08 20:43:08,464 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7> > > > [ID=8f88c6d5<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7>
> > [ID=8f88c6d5>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:40065<http://127.0.0.6:40065>]]
> > >
> > >
> > > HA “slave” broker log.
> > >
> > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > synchronized with live server,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > >
> > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221031: backup announced
> > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221084: Requested 0 quorum votes
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221071: Failing over based on quorum vote results.
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:51,946 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:56,977 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > destroyed
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2023-12-08 20:42:57,445 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6
> > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > > Retrying
> > > 2023-12-08 20:42:57,629 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > > [ID=483170f6<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:55911<http://127.0.0.6:55911>]]
> > > 2023-12-08 20:42:57,630 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > > [ID=4514a2b3<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:60199<http://127.0.0.6:60199>]]
> > > 2023-12-08 20:42:57,648 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > > [ID=fcde4d78<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:53221<http://127.0.0.6:53221>]]
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 29, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 31, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,708 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 33, result: OK
> > > 2023-12-08 20:42:57,709 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@3645037f
> > > 2023-12-08 20:42:57,719 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 35, result: OK
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:57,753 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > 2023-12-08 20:42:57,760 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 38, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, > > > with transaction: null
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 41, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 43, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 44, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 46, result: OK
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 48, result: OK
> > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > >
> > >
> > >
> >
>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
RE: Trouble with Replication HA Master/Slave config performing failback
Posted by John Lilley <jo...@redpointglobal.com.INVALID>.
Hi Paul,
Thanks for all of this! One thing you said struck me as odd: “You will need a separate PVC for both your Primary and standby broker instances.”
By “separate PVC” do you mean completely different volumes? Or a shared volume with a mount in each pod?
Does this mean that the primary and standby do not communicate anything via the lock file? Perhaps I am confused by the name “lock file” which implies some kind of concurrency management.
Instead of two separate PVC volumes, would it be OK to use a shared PVC (in Azure, this is an NFS-mounted FileShare volume) but set the primary and backup to use different directories within the volume? To save cost and complexity.
john
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Shields, Paul Michael <pa...@hpe.com>
Sent: Friday, January 19, 2024 9:48 AM
To: users@activemq.apache.org
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hi John,
I am using the activemq-artemis-operator from ArtemisCloud.io<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fArtemisCloud.io&c=E,1,TV8NB-Ny1beQqMowEfKkewBSKybapQAV_KnDp6P5OgtMCv0-n88PFErhGYbfjzDTXTShqF8_nA7PcwZSbhhhq9T1SeZtmTI-yE7OpN_U2KevLKqACVtBmbc,&typo=1&ancr_add=1> That operator only creates the PVC when I specify persistenceEnabled=true. So that is how I create a PVC for each broker instance. I then have to customize the broker.xml file to only have the <node-manager-lock-directory> on the PVC and the othe Artemis data directories on the local pod filesystem.
If you are deploying directly to k8s without an operator then you could have persistenceEnabled=false, but would have to define a PVC that would mount the <node-manager-lock-directory> to the broker pod to store the server.lock file. You will need a separate PVC for both your Primary and standby broker instances.
Yes, I have attached the broker.xml file to this email. Note that the only difference between my primary and standby instance broker.xml files is the <ha-policy> definition. So, I will attach the whole broker.xml from my Primary and just post the standby <ha-policy> portion.
Show the PVC mount:
ncn-m001:~ # kubectl -n dvs exec -it cray-dvs-mqtt-ss-0 -- /bin/bash
[jboss@cray-dvs-mqtt-ss-0 ~]$ df
Filesystem 1K-blocks Used Available Use% Mounted on
containerd_overlayfs 676576488 219342084 457234404 33% /
tmpfs 65536 0 65536 0% /dev
tmpfs 131773112 0 131773112 0% /sys/fs/cgroup
containerd_overlayfs 676576488 219342084 457234404 33% /etc/hostname
shm 65536 0 65536 0% /dev/shm
/dev/sdc3 187445840 222380 187223460 1% /etc/hosts
/dev/rbd2 1992552 34876 1941292 2% /opt/cray-dvs-mqtt/data <--- this is the PVC mount
tmpfs 263443828 4 263443824 1% /amq/extra/secrets/cray-dvs-mqtt-props
tmpfs 263443828 12 263443816 1% /run/secrets///kubernetes.io/serviceaccount<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f%2f%2fkubernetes.io%2fserviceaccount&c=E,1,zx-JtrSyBrNbA01-NTG91tZA5WOC-kl6BPtfG3RTuIQykLDhNJnVTq3pNbooNd06f7PobYG4wYRoE_WMe607UGKQe_yDrIGm_N6BvubSj6O0VVM,&typo=1&ancr_add=1>
tmpfs 131773112 0 131773112 0% /proc/acpi
tmpfs 131773112 0 131773112 0% /proc/scsi
tmpfs 131773112 0 131773112 0% /sys/firmware
Show the lock file on the PVC
[jboss@cray-dvs-mqtt-ss-0 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/server.lock
0000000 3050 fa30 65aa b57a 117b 9eee b25c bf25
0000020 7183 0096
0000023
[jboss@cray-dvs-mqtt-ss-0 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/serverlock.1
0000000
[jboss@cray-dvs-mqtt-ss-0 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/serverlock.2
0000000
[jboss@cray-dvs-mqtt-ss-0 ~]$ exit
exit
ncn-m001:~ # kubectl -n dvs exec -it cray-dvs-mqtt-ss-1 -- /bin/bash
[jboss@cray-dvs-mqtt-ss-1 ~]$ ls -l /opt/cray-dvs-mqtt/data/journal/
total 4
-rw-r--r-- 1 jboss root 19 Jan 17 22:26 server.lock
-rw-r--r-- 1 jboss root 0 Jan 17 22:25 serverlock.1
-rw-r--r-- 1 jboss root 0 Jan 17 22:25 serverlock.2
[jboss@cray-dvs-mqtt-ss-1 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/server.lock
0000000 3030 fa30 65aa b57a 117b 9eee b25c bf25
0000020 7183 0096
0000023
[jboss@cray-dvs-mqtt-ss-1 ~]$
Primary instance broker.xml file: Note I have logging enable. You probably don’t want that in a production environment.
[jboss@cray-dvs-mqtt-ss-0 ~]$ cat amq-broker/etc/broker.xml
<?xml version='1.0'?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<configuration xmlns="urn:activemq"
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
xmlns:xi=http://www.w3.org/2001/XInclude
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
xsi:schemaLocation="urn:activemq:core ">
<name>amq-broker</name>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<persistence-enabled>true</persistence-enabled>
<!-- this could be ASYNCIO, MAPPED, NIO
ASYNCIO: Linux Libaio
MAPPED: mmap files
NIO: Plain Java Files
-->
<journal-type>NIO</journal-type>
<node-manager-lock-directory>/opt/cray-dvs-mqtt/data/journal</node-manager-lock-directory>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<!-- if you want to retain your journal uncomment this following configuration.
This will allow your system to keep 7 days of your data, up to 10G. Tweak it accordingly to your use case and capacity.
it is recommended to use a separate storage unit from the journal for performance considerations.
<journal-retention-directory period="7" unit="DAYS" storage-limit="10G">data/retention</journal-retention-directory>
You can also enable retention by using the argument journal-retention on the `artemis create` command -->
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<!--
You can verify the network health of a particular NIC by specifying the <network-check-NIC> element.
<network-check-NIC>theNicName</network-check-NIC>
-->
<!--
Use this to use an HTTP server to validate the network
<network-check-URL-list>http://www.apache.org</network-check-URL-list<http://www.apache.org%3c/network-check-URL-list>> -->
<!-- <network-check-period>10000</network-check-period> -->
<!-- <network-check-timeout>1000</network-check-timeout> -->
<!-- this is a comma separated list, no spaces, just DNS or IPs
it should accept IPV6
Warning: Make sure you understand your network topology as this is meant to validate if your network is valid.
Using IPs that could eventually disappear or be partially visible may defeat the purpose.
You can use a list of multiple IPs, and if any successful ping will make the server OK to continue running -->
<!-- <network-check-list>10.0.0.1</network-check-list> -->
<!-- use this to customize the ping used for ipv4 addresses -->
<!-- <network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command> -->
<!-- use this to customize the ping used for ipv6 addresses -->
<!-- <network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command> -->
<connectors>
<!-- Connector used to be announced through cluster connections and notifications -->
<connector name="artemis">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616</connector>
</connectors>
<!-- how often we are looking for how many bytes are being used on the disk in ms -->
<disk-scan-period>5000</disk-scan-period>
<!-- once the disk hits this limit the system will block, or close the connection in certain protocols
that won't support flow control. -->
<max-disk-usage>90</max-disk-usage>
<!-- should the broker detect dead locks and other issues -->
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<!-- the system will enter into page mode once you hit this limit. This is an estimate in bytes of how much the messages are using in memory
The system will use half of the available memory (-Xmx) by default for the global-max-size.
You may specify a different value here if you need to customize it to your needs.
<global-max-size>100Mb</global-max-size> -->
<!-- the maximum number of messages accepted before entering full address mode.
if global-max-size is specified the full address mode will be specified by whatever hits it first. -->
<global-max-messages>-1</global-max-messages>
<acceptors>
<!-- useEpoll means: it will use Netty epoll if you are on a system (Linux) that supports it -->
<!-- amqpCredits: The number of credits sent to AMQP producers -->
<!-- amqpLowCredits: The server will send the # credits specified at amqpCredits at this low mark -->
<!-- amqpDuplicateDetection: If you are not using duplicate detection, set this to false
as duplicate detection requires applicationProperties to be parsed on the server. -->
<!-- amqpMinLargeMessageSize: Determines how many bytes are considered large, so we start using files to hold their data.
default: 102400, -1 would mean to disable large mesasge control -->
<!-- Note: If an acceptor needs to be compatible with HornetQ and/or Artemis 1.x clients add
"anycastPrefix=jms.queue.;multicastPrefix=jms.topic." to the acceptor url.
See https://issues.apache.org/jira/browse/ARTEMIS-1644 for more information. -->
<!-- Acceptor for every supported protocol -->
<acceptor name="dvs">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883?protocols=MQTT;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor><acceptor name="scaleDown">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616?protocols=CORE;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor>
</acceptors>
<cluster-user>EtBU3LYU</cluster-user>
<cluster-password>xOKqDzwH</cluster-password>
<broadcast-groups>
<broadcast-group name="my-broadcast-group"> <jgroups-file>jgroups-ping.xml</jgroups-file> <jgroups-channel>activemq_broadcast_channel</jgroups-channel> <connector-ref>artemis</connector-ref> </broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group"> <jgroups-file>jgroups-ping.xml</jgroups-file> <jgroups-channel>activemq_broadcast_channel</jgroups-channel> <refresh-timeout>10000</refresh-timeout> </discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster"> <connector-ref>artemis</connector-ref> <retry-interval>1000</retry-interval> <retry-interval-multiplier>2</retry-interval-multiplier> <max-retry-interval>32000</max-retry-interval> <initial-connect-attempts>20</initial-connect-attempts> <reconnect-attempts>-1</reconnect-attempts> <use-duplicate-detection>true</use-duplicate-detection> <message-load-balancing>ON_DEMAND</message-load-balancing> <max-hops>1</max-hops> <discovery-group-ref discovery-group-name="my-discovery-group"/> </cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="admin"/>
<permission type="deleteNonDurableQueue" roles="admin"/>
<permission type="createDurableQueue" roles="admin"/>
<permission type="deleteDurableQueue" roles="admin"/>
<permission type="createAddress" roles="admin"/>
<permission type="deleteAddress" roles="admin"/>
<permission type="consume" roles="admin"/>
<permission type="browse" roles="admin"/>
<permission type="send" roles="admin"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="admin"/>
</security-setting>
</security-settings>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="//activemq.management#"><https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f%2f%2factivemq.management%23%26quot%3b%26gt%3b&c=E,1,BRroUY8kfUeX80gvrp7t72VVHVGMPASjpLV9yYkz4JVee8GpRfZvI_Kw2UOOUNV2uD0jK5gOPSY2kMhPWHhNzPMAHND9p06dXyXm9NI42fIRLB9HOfs,&typo=1&ancr_add=1>
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<redistribution-delay>0</redistribution-delay>
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- if max-size-bytes and max-size-messages were both enabled, the system will enter into paging
based on the first attribute to hits the maximum value -->
<!-- limit for the address in bytes, -1 means unlimited -->
<max-size-bytes>-1</max-size-bytes>
<!-- limit for the address in messages, -1 means unlimited -->
<max-size-messages>-1</max-size-messages>
<!-- the size of each file on paging. Notice we keep files in memory while they are in use.
Lower this setting if you have too many queues in memory. -->
<page-size-bytes>10M</page-size-bytes>
<!-- limit how many messages are read from paging into the Queue. -->
<max-read-page-messages>-1</max-read-page-messages>
<!-- limit how much memory is read from paging into the Queue. -->
<max-read-page-bytes>20M</max-read-page-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-delete-queues>false</auto-delete-queues>
<auto-delete-addresses>false</auto-delete-addresses>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
<broker-plugins>
<broker-plugin class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
<property key="LOG_ALL_EVENTS" value="true"/>
<property key="LOG_CONNECTION_EVENTS" value="true"/>
<property key="LOG_SESSION_EVENTS" value="true"/>
<property key="LOG_CONSUMER_EVENTS" value="true"/>
<property key="LOG_DELIVERING_EVENTS" value="true"/>
<property key="LOG_SENDING_EVENTS" value="true"/>
<property key="LOG_INTERNAL_EVENTS" value="true"/>
</broker-plugin>
</broker-plugins>
<metrics> <plugin class-name="com.redhat.amq.broker.core.server.metrics.plugins.ArtemisPrometheusMetricsPlugin"/> </metrics>
</core>
</configuration>
[jboss@cray-dvs-mqtt-ss-0 ~]$
Standby instance broker.xml <ha-policy> stanza
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Date: Friday, January 19, 2024 at 10:21 AM
To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
Cc: Lino Pereira <li...@redpointglobal.com>>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Paul,
Thanks for the information! It sounds like you have experience with AMQ in K8S. Can you share your two broker.xml files with us?
Lino: do we currently set persistenceEnable=false? I would expect there’s no reason for setting it true, since the pod-local storage will evaporate with the pod.
Thanks
John
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,_MEdc5Qy-vTEnjozH--yv7M1N-lLpEcvjRVTe0TZfe8G_dcxUJzpNCwemRvf2WA7KwCsV-QQ_ejGemmADMP_e4L0WvHHBpjTniOeKgAhT3d5l10,&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Shields, Paul Michael <pa...@hpe.com>>
Sent: Friday, January 19, 2024 8:51 AM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hi John,
The <node-manager-lock-directory>. Needs to be specified in both the master and slave broker.xml files.
the nodeUUID which is contained in the server lock file was changing between active broker startups. The server.lock file is stored in the /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our current configuration of the broker has persistenceEnable=false which causes any data/broker state to be removed when the broker pod is destroyed. Thus a new nodeUUID was being generated on each Primary broker startup. When the Primary broker instance starts up it looks for an existing server.lock file. When it does not find one it generates a new nodeUUID and places it in the server.lock file, otherwise it uses the nodeUUID contained within the existing server.lock file. The Primary broker then communicates this nodeUUID through the cluster connection to the Standby broker instance on startup. The Standby broker instance uses the nodeUUID to determine if the Primary broker instance is part of its HA pair. The Standby broker instance saved off the original nodeUUID when the HA cluster was first started and since the recently communicated nodeUUID does not match the nodeUUID that it previously stored the fail back is not performed. Once persistence was enabled the Standby broker instance performed the fallback to the Primary instance and was no longer in the "active" state
This is what I have observed in our Kubernetes environment. Justin please make any corrections to my description of the nodeUUID use.
Paul
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Date: Thursday, January 18, 2024 at 5:49 PM
To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Justin,
Can you elaborate on the lock file in the <node-manager-lock-directory>?
Does it need to be specified in the broker.xml for both the master and the slave?
Does it actually get used as a “lock” in the concurrency sense, to coordinate the live/backup election?
Thanks
john
[rg]<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,0rYuECbqGwpo9IjFKJrQQrfyzL5V_aEiqSXV25I7DeF4wuQn3I6z1sJCY1NkNJ-bjdpV8nVix_VDU0mK5tgrxuSQaSyrlVLxE5EiTf0Z8G4F&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Monday, January 15, 2024 9:56 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
In the replication case you still need a PV in K8s, but I don't believe you need to put the whole journal on the PV (and suffer the performance penalty). You should just need to point the node-manager-lock-directory in broker.xml to the PV.
Justin
On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
Based on your reply in the provided link, are you saying that even for the replication HA case, you still need an external FileShare PV, in K8s, so that the nodeID can be persisted between restarts. In this use case, is the performance of this FileShare less of a concern than it is for the shared-store use case?
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
-----Original Message-----
From: Justin Bertram <jb...@apache.org>>
Sent: Friday, January 12, 2024 4:53 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
I sent that email on December 15. You can find it on the web-based mailing
list interface [1].
Justin
[1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l
On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <pa...@hpe.com>>
wrote:
> Hi Justin,
>
> Forgive me but I don’t remember a journal persistence issue discussion. I
> did an email search too and did not see that discussion either. Would you
> please refresh me on the discussion? I may have inadvertently deleted that?
>
> I do see data being replicated to my production slave/standby instance on
> initial startup of the two broker instances. Failover works as expected.
> But when the primary instance starts backup I do not see the active slave
> replicate to the starting primary and I do not see the active slave restart
> and go to the backup server state.
>
> What is the communication sequence when the primary starts up after it
> quit with the active backup broker? In other words, how is the active
> backup broker notified to start replication to the newly started primary
> broker?
>
> I ran the replicated-failback example again with TRACE level logging
> turned on and I do see the attemptFailback=false message in the
> target/server1/log/artemis.log file at startup, so it must be changed later
> or am I not understanding the purpose of this flag?
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> discoveryInitialWaitTimeout=10000}]
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost, b=null],
> nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost)
>
> From my production slave broker log
>
>
> 024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> Thanks,
> Paul
> From: Justin Bertram <jb...@apache.org>>
> Date: Friday, January 12, 2024 at 3:38 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > Is it possible to use jgroups instead of UDP for the cluster connection
> in the example?
>
> Yes. Check out examples/features/clustered/clustered-jgroups for an example
> configuration. Keep in mind that UDP, JGroups, etc. is only used for
> initial discovery. Once brokers are discovered then normal TCP connections
> are created.
>
> Before I continue looking through your logs, etc. can you confirm that you
> resolved the journal persistence issue I described in my previous email? If
> that's not resolved there's no chance for failback to occur.
>
>
> Justin
>
>
>
> On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>>
> wrote:
>
> > Hi Justin,
> >
> > I tried the example and it works. I turned up logging to get a success
> > path through the code to maybe see where things get de-railed. Is it
> > possible to use jgroups instead of UDP for the cluster connection in the
> > example? That would give me a better success signature.
> >
> > My standby instance has this in the broker.xml which is the same as in
> the
> > example:
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > I am seeing this series of log messages that suggests that failback is
> not
> > enabled. Is the attemptFailback=false the default at this point in
> startup
> > and it is changed later or it there something gone wrong at this point?
> >
> > 2024-01-11 18:24:14,066 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > QuorumManager(server=null) about node up =
> > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> > <mailto:
<mailto:%0b>> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> >
> > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > ClientSessionFactoryImpl received backup update for live/backup pair =
> > TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > / null but it didn't belong to TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,067 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Cluster Connected
> >
> > 2024-01-11 18:24:14,068 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting backup manager
> >
> > 2024-01-11 18:24:14,069 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > backup config ClusterConnectionConfiguration{name='my-cluster',
> address='',
> > connectorName='artemis', clientFailureCheckPeriod=30000,
> > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > clusterNotificationAttempts=2}
> >
> > 2024-01-11 18:24:14,081 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > 2024-01-11 18:24:14,083 TRACE
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> >
> > java.lang.Exception:trace<ftp://java.lang.Exception/trace>
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4 rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying reconnection attempt 0/0
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18> > > <mailto:connectorFactory<mailto:connectorFactory>
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18>
> >
> > and currentConnectorConfig: TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Connector NettyConnector
> > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using native epoll
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Started EPOLL Netty Connector version 4.1.86.Final to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Remote destination:
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.25:61616<ftp://10.44.0.25/61616>
> >
> >
> > Thanks,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>>
> > Date: Wednesday, December 13, 2023 at 3:23 PM
> > To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > ActiveMQ Artemis ships with an example in the
> > examples/features/ha/replicated-failback directory that demonstrates how
> > this works. I ran this example and it worked fine. I also started up the
> > brokers manually and triggered failover and failback. Again, everything
> > worked fine. My only guess is that there's something configured wrong or
> > perhaps there's an environmental issue that's preventing failback from
> > working properly for you. If you could provide clear steps on how to
> > reproduce this issue I could investigate further.
> >
> >
> > Justin
> >
> > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am having an issue with failback to the master broker. I start the
> > > master broker then the slave broker. I see the replication from the
> > master
> > > to the slave happen. I stop the master/active broker and a failover
> > happens
> > > to the slave as expected. When the master is started a failback to it
> is
> > > expected but does not happen. The following is my ha-policy config.
> > Did I
> > > miss something in the ha-policy configuration? We are using version
> > 2.28.0
> > > and java17.
> > >
> > > Thanks,
> > > Paul
> > >
> > >
> > > Master broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <master>
> > >
> > > <check-for-live-server>true</check-for-live-server>
> > >
> > > </master>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > Slave broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > The following are the logs from the master and slave brokers
> > > HA “master” broker log after restart
> > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > Starting embedded web server
> > > 2023-12-08 20:42:57,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 15, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,627 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3
> > > 2023-12-08 20:42:57,657 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2> > > > [ID=47edf16f<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2>
> > [ID=47edf16f>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/10.44.0.18:34508<http://10.44.0.18:34508>]]
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > > activemq-branding plugin
> > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > artemis-plugin plugin
> > > 2023-12-08 20:42:57,766 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 18, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,767 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 20, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > Initialising
> > > hawtio services
> > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > Configuration
> > > will be discovered via system properties
> > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > Hawtio 2.15.0
> > > 2023-12-08 20:42:57,975 INFO
> > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > authentication filter, JAAS realm: "activemq" authorized role(s):
> "admin"
> > > role principal classes:
> > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > servlet is disabled
> > > 2023-12-08 20:42:57,989 INFO
> > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > > property: [key=policyLocation,
> > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > HTTP Server started at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > >
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > Artemis Jolokia REST API available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > >
> > >
> > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > Console
> > > available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> > >
> > > 20:42:58,455 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl]
> > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d> > > > [ID=f85ec51b<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d>
> > [ID=f85ec51b>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:49865<http://127.0.0.6:49865>]]
> > > 2023-12-08 20:42:59,524 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858> > > > [ID=7cef23a2<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858>
> > [ID=7cef23a2>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:59691<http://127.0.0.6:59691>]]
> > > 2023-12-08 20:42:59,535 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 22, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,536 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,549 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 24, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,550 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,583 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: OK
> > > 2023-12-08 20:42:59,584 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@5f46053b
> > > 2023-12-08 20:42:59,593 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > 2023-12-08 20:42:59,594 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 28, result: OK
> > > 2023-12-08 20:42:59,616 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:59,615 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 31, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 32, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 35, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 39, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:43:08,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256> > > > [ID=1d1d79ce<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256>
> > [ID=1d1d79ce>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:60787<http://127.0.0.6:60787>]]
> > > 2023-12-08 20:43:08,464 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7> > > > [ID=8f88c6d5<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7>
> > [ID=8f88c6d5>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:40065<http://127.0.0.6:40065>]]
> > >
> > >
> > > HA “slave” broker log.
> > >
> > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > synchronized with live server,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > >
> > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221031: backup announced
> > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221084: Requested 0 quorum votes
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221071: Failing over based on quorum vote results.
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:51,946 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:56,977 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > destroyed
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2023-12-08 20:42:57,445 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6
> > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > > Retrying
> > > 2023-12-08 20:42:57,629 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > > [ID=483170f6<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:55911<http://127.0.0.6:55911>]]
> > > 2023-12-08 20:42:57,630 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > > [ID=4514a2b3<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:60199<http://127.0.0.6:60199>]]
> > > 2023-12-08 20:42:57,648 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > > [ID=fcde4d78<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:53221<http://127.0.0.6:53221>]]
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 29, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 31, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,708 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 33, result: OK
> > > 2023-12-08 20:42:57,709 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@3645037f
> > > 2023-12-08 20:42:57,719 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 35, result: OK
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:57,753 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > 2023-12-08 20:42:57,760 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 38, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, > > > with transaction: null
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 41, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 43, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 44, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 46, result: OK
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 48, result: OK
> > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > >
> > >
> > >
> >
>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by "Shields, Paul Michael" <pa...@hpe.com>.
Hi John,
I am using the activemq-artemis-operator from ArtemisCloud.io. That operator only creates the PVC when I specify persistenceEnabled=true. So that is how I create a PVC for each broker instance. I then have to customize the broker.xml file to only have the <node-manager-lock-directory> on the PVC and the othe Artemis data directories on the local pod filesystem.
If you are deploying directly to k8s without an operator then you could have persistenceEnabled=false, but would have to define a PVC that would mount the <node-manager-lock-directory> to the broker pod to store the server.lock file. You will need a separate PVC for both your Primary and standby broker instances.
Yes, I have attached the broker.xml file to this email. Note that the only difference between my primary and standby instance broker.xml files is the <ha-policy> definition. So, I will attach the whole broker.xml from my Primary and just post the standby <ha-policy> portion.
Show the PVC mount:
ncn-m001:~ # kubectl -n dvs exec -it cray-dvs-mqtt-ss-0 -- /bin/bash
[jboss@cray-dvs-mqtt-ss-0 ~]$ df
Filesystem 1K-blocks Used Available Use% Mounted on
containerd_overlayfs 676576488 219342084 457234404 33% /
tmpfs 65536 0 65536 0% /dev
tmpfs 131773112 0 131773112 0% /sys/fs/cgroup
containerd_overlayfs 676576488 219342084 457234404 33% /etc/hostname
shm 65536 0 65536 0% /dev/shm
/dev/sdc3 187445840 222380 187223460 1% /etc/hosts
/dev/rbd2 1992552 34876 1941292 2% /opt/cray-dvs-mqtt/data •- this is the PVC mount
tmpfs 263443828 4 263443824 1% /amq/extra/secrets/cray-dvs-mqtt-props
tmpfs 263443828 12 263443816 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 131773112 0 131773112 0% /proc/acpi
tmpfs 131773112 0 131773112 0% /proc/scsi
tmpfs 131773112 0 131773112 0% /sys/firmware
Show the lock file on the PVC
[jboss@cray-dvs-mqtt-ss-0 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/server.lock
0000000 3050 fa30 65aa b57a 117b 9eee b25c bf25
0000020 7183 0096
0000023
[jboss@cray-dvs-mqtt-ss-0 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/serverlock.1
0000000
[jboss@cray-dvs-mqtt-ss-0 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/serverlock.2
0000000
[jboss@cray-dvs-mqtt-ss-0 ~]$ exit
exit
ncn-m001:~ # kubectl -n dvs exec -it cray-dvs-mqtt-ss-1 -- /bin/bash
[jboss@cray-dvs-mqtt-ss-1 ~]$ ls -l /opt/cray-dvs-mqtt/data/journal/
total 4
-rw-r--r-- 1 jboss root 19 Jan 17 22:26 server.lock
-rw-r--r-- 1 jboss root 0 Jan 17 22:25 serverlock.1
-rw-r--r-- 1 jboss root 0 Jan 17 22:25 serverlock.2
[jboss@cray-dvs-mqtt-ss-1 ~]$ od -x /opt/cray-dvs-mqtt/data/journal/server.lock
0000000 3030 fa30 65aa b57a 117b 9eee b25c bf25
0000020 7183 0096
0000023
[jboss@cray-dvs-mqtt-ss-1 ~]$
Primary instance broker.xml file: Note I have logging enable. You probably don’t want that in a production environment.
[jboss@cray-dvs-mqtt-ss-0 ~]$ cat amq-broker/etc/broker.xml
<?xml version='1.0'?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<configuration xmlns="urn:activemq"
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
xmlns:xi=http://www.w3.org/2001/XInclude
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
xsi:schemaLocation="urn:activemq:core ">
<name>amq-broker</name>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<persistence-enabled>true</persistence-enabled>
<!-- this could be ASYNCIO, MAPPED, NIO
ASYNCIO: Linux Libaio
MAPPED: mmap files
NIO: Plain Java Files
-->
<journal-type>NIO</journal-type>
<node-manager-lock-directory>/opt/cray-dvs-mqtt/data/journal</node-manager-lock-directory>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<!-- if you want to retain your journal uncomment this following configuration.
This will allow your system to keep 7 days of your data, up to 10G. Tweak it accordingly to your use case and capacity.
it is recommended to use a separate storage unit from the journal for performance considerations.
<journal-retention-directory period="7" unit="DAYS" storage-limit="10G">data/retention</journal-retention-directory>
You can also enable retention by using the argument journal-retention on the `artemis create` command -->
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<!--
You can verify the network health of a particular NIC by specifying the <network-check-NIC> element.
<network-check-NIC>theNicName</network-check-NIC>
-->
<!--
Use this to use an HTTP server to validate the network
<network-check-URL-list>http://www.apache.org</network-check-URL-list<http://www.apache.org%3c/network-check-URL-list>> -->
<!-- <network-check-period>10000</network-check-period> -->
<!-- <network-check-timeout>1000</network-check-timeout> -->
<!-- this is a comma separated list, no spaces, just DNS or IPs
it should accept IPV6
Warning: Make sure you understand your network topology as this is meant to validate if your network is valid.
Using IPs that could eventually disappear or be partially visible may defeat the purpose.
You can use a list of multiple IPs, and if any successful ping will make the server OK to continue running -->
<!-- <network-check-list>10.0.0.1</network-check-list> -->
<!-- use this to customize the ping used for ipv4 addresses -->
<!-- <network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command> -->
<!-- use this to customize the ping used for ipv6 addresses -->
<!-- <network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command> -->
<connectors>
<!-- Connector used to be announced through cluster connections and notifications -->
<connector name="artemis">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616</connector>
</connectors>
<!-- how often we are looking for how many bytes are being used on the disk in ms -->
<disk-scan-period>5000</disk-scan-period>
<!-- once the disk hits this limit the system will block, or close the connection in certain protocols
that won't support flow control. -->
<max-disk-usage>90</max-disk-usage>
<!-- should the broker detect dead locks and other issues -->
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<!-- the system will enter into page mode once you hit this limit. This is an estimate in bytes of how much the messages are using in memory
The system will use half of the available memory (-Xmx) by default for the global-max-size.
You may specify a different value here if you need to customize it to your needs.
<global-max-size>100Mb</global-max-size> -->
<!-- the maximum number of messages accepted before entering full address mode.
if global-max-size is specified the full address mode will be specified by whatever hits it first. -->
<global-max-messages>-1</global-max-messages>
<acceptors>
<!-- useEpoll means: it will use Netty epoll if you are on a system (Linux) that supports it -->
<!-- amqpCredits: The number of credits sent to AMQP producers -->
<!-- amqpLowCredits: The server will send the # credits specified at amqpCredits at this low mark -->
<!-- amqpDuplicateDetection: If you are not using duplicate detection, set this to false
as duplicate detection requires applicationProperties to be parsed on the server. -->
<!-- amqpMinLargeMessageSize: Determines how many bytes are considered large, so we start using files to hold their data.
default: 102400, -1 would mean to disable large mesasge control -->
<!-- Note: If an acceptor needs to be compatible with HornetQ and/or Artemis 1.x clients add
"anycastPrefix=jms.queue.;multicastPrefix=jms.topic." to the acceptor url.
See https://issues.apache.org/jira/browse/ARTEMIS-1644 for more information. -->
<!-- Acceptor for every supported protocol -->
<acceptor name="dvs">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883?protocols=MQTT;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor><acceptor name="scaleDown">tcp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616?protocols=CORE;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor>
</acceptors>
<cluster-user>EtBU3LYU</cluster-user>
<cluster-password>xOKqDzwH</cluster-password>
<broadcast-groups>
<broadcast-group name="my-broadcast-group"> <jgroups-file>jgroups-ping.xml</jgroups-file> <jgroups-channel>activemq_broadcast_channel</jgroups-channel> <connector-ref>artemis</connector-ref> </broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group"> <jgroups-file>jgroups-ping.xml</jgroups-file> <jgroups-channel>activemq_broadcast_channel</jgroups-channel> <refresh-timeout>10000</refresh-timeout> </discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster"> <connector-ref>artemis</connector-ref> <retry-interval>1000</retry-interval> <retry-interval-multiplier>2</retry-interval-multiplier> <max-retry-interval>32000</max-retry-interval> <initial-connect-attempts>20</initial-connect-attempts> <reconnect-attempts>-1</reconnect-attempts> <use-duplicate-detection>true</use-duplicate-detection> <message-load-balancing>ON_DEMAND</message-load-balancing> <max-hops>1</max-hops> <discovery-group-ref discovery-group-name="my-discovery-group"/> </cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="admin"/>
<permission type="deleteNonDurableQueue" roles="admin"/>
<permission type="createDurableQueue" roles="admin"/>
<permission type="deleteDurableQueue" roles="admin"/>
<permission type="createAddress" roles="admin"/>
<permission type="deleteAddress" roles="admin"/>
<permission type="consume" roles="admin"/>
<permission type="browse" roles="admin"/>
<permission type="send" roles="admin"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="admin"/>
</security-setting>
</security-settings>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<redistribution-delay>0</redistribution-delay>
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- if max-size-bytes and max-size-messages were both enabled, the system will enter into paging
based on the first attribute to hits the maximum value -->
<!-- limit for the address in bytes, -1 means unlimited -->
<max-size-bytes>-1</max-size-bytes>
<!-- limit for the address in messages, -1 means unlimited -->
<max-size-messages>-1</max-size-messages>
<!-- the size of each file on paging. Notice we keep files in memory while they are in use.
Lower this setting if you have too many queues in memory. -->
<page-size-bytes>10M</page-size-bytes>
<!-- limit how many messages are read from paging into the Queue. -->
<max-read-page-messages>-1</max-read-page-messages>
<!-- limit how much memory is read from paging into the Queue. -->
<max-read-page-bytes>20M</max-read-page-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-delete-queues>false</auto-delete-queues>
<auto-delete-addresses>false</auto-delete-addresses>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
<broker-plugins>
<broker-plugin class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
<property key="LOG_ALL_EVENTS" value="true"/>
<property key="LOG_CONNECTION_EVENTS" value="true"/>
<property key="LOG_SESSION_EVENTS" value="true"/>
<property key="LOG_CONSUMER_EVENTS" value="true"/>
<property key="LOG_DELIVERING_EVENTS" value="true"/>
<property key="LOG_SENDING_EVENTS" value="true"/>
<property key="LOG_INTERNAL_EVENTS" value="true"/>
</broker-plugin>
</broker-plugins>
<metrics> <plugin class-name="com.redhat.amq.broker.core.server.metrics.plugins.ArtemisPrometheusMetricsPlugin"/> </metrics>
</core>
</configuration>
[jboss@cray-dvs-mqtt-ss-0 ~]$
Standby instance broker.xml <ha-policy> stanza
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
From: John Lilley <jo...@redpointglobal.com.INVALID>
Date: Friday, January 19, 2024 at 10:21 AM
To: users@activemq.apache.org <us...@activemq.apache.org>
Cc: Lino Pereira <li...@redpointglobal.com>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Paul,
Thanks for the information! It sounds like you have experience with AMQ in K8S. Can you share your two broker.xml files with us?
Lino: do we currently set persistenceEnable=false? I would expect there’s no reason for setting it true, since the pod-local storage will evaporate with the pod.
Thanks
John
[rg]<https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Shields, Paul Michael <pa...@hpe.com>
Sent: Friday, January 19, 2024 8:51 AM
To: users@activemq.apache.org
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hi John,
The <node-manager-lock-directory>. Needs to be specified in both the master and slave broker.xml files.
the nodeUUID which is contained in the server lock file was changing between active broker startups. The server.lock file is stored in the /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our current configuration of the broker has persistenceEnable=false which causes any data/broker state to be removed when the broker pod is destroyed. Thus a new nodeUUID was being generated on each Primary broker startup. When the Primary broker instance starts up it looks for an existing server.lock file. When it does not find one it generates a new nodeUUID and places it in the server.lock file, otherwise it uses the nodeUUID contained within the existing server.lock file. The Primary broker then communicates this nodeUUID through the cluster connection to the Standby broker instance on startup. The Standby broker instance uses the nodeUUID to determine if the Primary broker instance is part of its HA pair. The Standby broker instance saved off the original nodeUUID when the HA cluster was first started and since the recently communicated nodeUUID does not match the nodeUUID that it previously stored the fail back is not performed. Once persistence was enabled the Standby broker instance performed the fallback to the Primary instance and was no longer in the "active" state
This is what I have observed in our Kubernetes environment. Justin please make any corrections to my description of the nodeUUID use.
Paul
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Date: Thursday, January 18, 2024 at 5:49 PM
To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Justin,
Can you elaborate on the lock file in the <node-manager-lock-directory>?
Does it need to be specified in the broker.xml for both the master and the slave?
Does it actually get used as a “lock” in the concurrency sense, to coordinate the live/backup election?
Thanks
john
[rg]<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,0rYuECbqGwpo9IjFKJrQQrfyzL5V_aEiqSXV25I7DeF4wuQn3I6z1sJCY1NkNJ-bjdpV8nVix_VDU0mK5tgrxuSQaSyrlVLxE5EiTf0Z8G4F&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Monday, January 15, 2024 9:56 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
In the replication case you still need a PV in K8s, but I don't believe you need to put the whole journal on the PV (and suffer the performance penalty). You should just need to point the node-manager-lock-directory in broker.xml to the PV.
Justin
On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
Based on your reply in the provided link, are you saying that even for the replication HA case, you still need an external FileShare PV, in K8s, so that the nodeID can be persisted between restarts. In this use case, is the performance of this FileShare less of a concern than it is for the shared-store use case?
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
-----Original Message-----
From: Justin Bertram <jb...@apache.org>>
Sent: Friday, January 12, 2024 4:53 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
I sent that email on December 15. You can find it on the web-based mailing
list interface [1].
Justin
[1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l<https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l>
On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <pa...@hpe.com>>
wrote:
> Hi Justin,
>
> Forgive me but I don’t remember a journal persistence issue discussion. I
> did an email search too and did not see that discussion either. Would you
> please refresh me on the discussion? I may have inadvertently deleted that?
>
> I do see data being replicated to my production slave/standby instance on
> initial startup of the two broker instances. Failover works as expected.
> But when the primary instance starts backup I do not see the active slave
> replicate to the starting primary and I do not see the active slave restart
> and go to the backup server state.
>
> What is the communication sequence when the primary starts up after it
> quit with the active backup broker? In other words, how is the active
> backup broker notified to start replication to the newly started primary
> broker?
>
> I ran the replicated-failback example again with TRACE level logging
> turned on and I do see the attemptFailback=false message in the
> target/server1/log/artemis.log file at startup, so it must be changed later
> or am I not understanding the purpose of this flag?
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> discoveryInitialWaitTimeout=10000}]
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost, b=null],
> nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost)
>
> From my production slave broker log
>
>
> 024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> Thanks,
> Paul
> From: Justin Bertram <jb...@apache.org>>
> Date: Friday, January 12, 2024 at 3:38 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > Is it possible to use jgroups instead of UDP for the cluster connection
> in the example?
>
> Yes. Check out examples/features/clustered/clustered-jgroups for an example
> configuration. Keep in mind that UDP, JGroups, etc. is only used for
> initial discovery. Once brokers are discovered then normal TCP connections
> are created.
>
> Before I continue looking through your logs, etc. can you confirm that you
> resolved the journal persistence issue I described in my previous email? If
> that's not resolved there's no chance for failback to occur.
>
>
> Justin
>
>
>
> On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>>
> wrote:
>
> > Hi Justin,
> >
> > I tried the example and it works. I turned up logging to get a success
> > path through the code to maybe see where things get de-railed. Is it
> > possible to use jgroups instead of UDP for the cluster connection in the
> > example? That would give me a better success signature.
> >
> > My standby instance has this in the broker.xml which is the same as in
> the
> > example:
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > I am seeing this series of log messages that suggests that failback is
> not
> > enabled. Is the attemptFailback=false the default at this point in
> startup
> > and it is changed later or it there something gone wrong at this point?
> >
> > 2024-01-11 18:24:14,066 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > QuorumManager(server=null) about node up =
> > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> > <mailto:
<mailto:%0b>> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> >
> > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > ClientSessionFactoryImpl received backup update for live/backup pair =
> > TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > / null but it didn't belong to TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,067 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Cluster Connected
> >
> > 2024-01-11 18:24:14,068 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting backup manager
> >
> > 2024-01-11 18:24:14,069 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > backup config ClusterConnectionConfiguration{name='my-cluster',
> address='',
> > connectorName='artemis', clientFailureCheckPeriod=30000,
> > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > clusterNotificationAttempts=2}
> >
> > 2024-01-11 18:24:14,081 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > 2024-01-11 18:24:14,083 TRACE
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> >
> > java.lang.Exception:trace<ftp://java.lang.Exception/trace>
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4 rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying reconnection attempt 0/0
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18> > > <mailto:connectorFactory<mailto:connectorFactory>
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18>
> >
> > and currentConnectorConfig: TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Connector NettyConnector
> > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using native epoll
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Started EPOLL Netty Connector version 4.1.86.Final to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Remote destination:
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.25:61616<ftp://10.44.0.25/61616>
> >
> >
> > Thanks,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>>
> > Date: Wednesday, December 13, 2023 at 3:23 PM
> > To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > ActiveMQ Artemis ships with an example in the
> > examples/features/ha/replicated-failback directory that demonstrates how
> > this works. I ran this example and it worked fine. I also started up the
> > brokers manually and triggered failover and failback. Again, everything
> > worked fine. My only guess is that there's something configured wrong or
> > perhaps there's an environmental issue that's preventing failback from
> > working properly for you. If you could provide clear steps on how to
> > reproduce this issue I could investigate further.
> >
> >
> > Justin
> >
> > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am having an issue with failback to the master broker. I start the
> > > master broker then the slave broker. I see the replication from the
> > master
> > > to the slave happen. I stop the master/active broker and a failover
> > happens
> > > to the slave as expected. When the master is started a failback to it
> is
> > > expected but does not happen. The following is my ha-policy config.
> > Did I
> > > miss something in the ha-policy configuration? We are using version
> > 2.28.0
> > > and java17.
> > >
> > > Thanks,
> > > Paul
> > >
> > >
> > > Master broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <master>
> > >
> > > <check-for-live-server>true</check-for-live-server>
> > >
> > > </master>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > Slave broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > The following are the logs from the master and slave brokers
> > > HA “master” broker log after restart
> > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > Starting embedded web server
> > > 2023-12-08 20:42:57,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 15, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,627 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3
> > > 2023-12-08 20:42:57,657 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2> > > > [ID=47edf16f<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2>
> > [ID=47edf16f>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/10.44.0.18:34508<http://10.44.0.18:34508>]]
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > > activemq-branding plugin
> > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > artemis-plugin plugin
> > > 2023-12-08 20:42:57,766 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 18, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,767 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 20, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > Initialising
> > > hawtio services
> > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > Configuration
> > > will be discovered via system properties
> > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > Hawtio 2.15.0
> > > 2023-12-08 20:42:57,975 INFO
> > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > authentication filter, JAAS realm: "activemq" authorized role(s):
> "admin"
> > > role principal classes:
> > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > servlet is disabled
> > > 2023-12-08 20:42:57,989 INFO
> > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > > property: [key=policyLocation,
> > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > HTTP Server started at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> >
> > >
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > Artemis Jolokia REST API available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08> > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08> > >
> > >
> > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > Console
> > > available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08>
> >
> > >
> > > 20:42:58,455 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl]
> > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d> > > > [ID=f85ec51b<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d>
> > [ID=f85ec51b>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:49865<http://127.0.0.6:49865>]]
> > > 2023-12-08 20:42:59,524 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858> > > > [ID=7cef23a2<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858>
> > [ID=7cef23a2>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:59691<http://127.0.0.6:59691>]]
> > > 2023-12-08 20:42:59,535 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 22, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,536 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,549 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 24, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,550 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,583 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: OK
> > > 2023-12-08 20:42:59,584 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@5f46053b
> > > 2023-12-08 20:42:59,593 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > 2023-12-08 20:42:59,594 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 28, result: OK
> > > 2023-12-08 20:42:59,616 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:59,615 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 31, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 32, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 35, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 39, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:43:08,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256> > > > [ID=1d1d79ce<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256>
> > [ID=1d1d79ce>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:60787<http://127.0.0.6:60787>]]
> > > 2023-12-08 20:43:08,464 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7> > > > [ID=8f88c6d5<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7>
> > [ID=8f88c6d5>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:40065<http://127.0.0.6:40065>]]
> > >
> > >
> > > HA “slave” broker log.
> > >
> > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > synchronized with live server,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > >
> > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221031: backup announced
> > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221084: Requested 0 quorum votes
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221071: Failing over based on quorum vote results.
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:51,946 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:56,977 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > destroyed
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2023-12-08 20:42:57,445 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6
> > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > > Retrying
> > > 2023-12-08 20:42:57,629 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > > [ID=483170f6<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:55911<http://127.0.0.6:55911>]]
> > > 2023-12-08 20:42:57,630 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > > [ID=4514a2b3<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:60199<http://127.0.0.6:60199>]]
> > > 2023-12-08 20:42:57,648 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > > [ID=fcde4d78<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:53221<http://127.0.0.6:53221>]]
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 29, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 31, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,708 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 33, result: OK
> > > 2023-12-08 20:42:57,709 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@3645037f
> > > 2023-12-08 20:42:57,719 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 35, result: OK
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:57,753 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > 2023-12-08 20:42:57,760 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 38, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, > > > with transaction: null
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 41, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 43, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 44, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 46, result: OK
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 48, result: OK
> > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > >
> > >
> > >
> >
>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
RE: Trouble with Replication HA Master/Slave config performing failback
Posted by Lino Pereira <li...@redpointglobal.com.INVALID>.
Thanks Paul.
John, we do set persistenceEnable=true because I though it was required for replication, be it short lived.
We also set max-saved-replicated-journals-size=0 on the slave, which tells the backup not to restart after failback as there will be > 0 backups saved. This was taken from one of the examples.
Thanks,
Lino
[rg] <https://www.redpointglobal.com/>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: John Lilley <jo...@redpointglobal.com>
Sent: Friday, January 19, 2024 9:20 AM
To: users@activemq.apache.org
Cc: Lino Pereira <li...@redpointglobal.com>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Paul,
Thanks for the information! It sounds like you have experience with AMQ in K8S. Can you share your two broker.xml files with us?
Lino: do we currently set persistenceEnable=false? I would expect there’s no reason for setting it true, since the pod-local storage will evaporate with the pod.
Thanks
John
From: Shields, Paul Michael <pa...@hpe.com>>
Sent: Friday, January 19, 2024 8:51 AM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hi John,
The <node-manager-lock-directory>. Needs to be specified in both the master and slave broker.xml files.
the nodeUUID which is contained in the server lock file was changing between active broker startups. The server.lock file is stored in the /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our current configuration of the broker has persistenceEnable=false which causes any data/broker state to be removed when the broker pod is destroyed. Thus a new nodeUUID was being generated on each Primary broker startup. When the Primary broker instance starts up it looks for an existing server.lock file. When it does not find one it generates a new nodeUUID and places it in the server.lock file, otherwise it uses the nodeUUID contained within the existing server.lock file. The Primary broker then communicates this nodeUUID through the cluster connection to the Standby broker instance on startup. The Standby broker instance uses the nodeUUID to determine if the Primary broker instance is part of its HA pair. The Standby broker instance saved off the original nodeUUID when the HA cluster was first started and since the recently communicated nodeUUID does not match the nodeUUID that it previously stored the fail back is not performed. Once persistence was enabled the Standby broker instance performed the fallback to the Primary instance and was no longer in the "active" state
This is what I have observed in our Kubernetes environment. Justin please make any corrections to my description of the nodeUUID use.
Paul
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Date: Thursday, January 18, 2024 at 5:49 PM
To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Justin,
Can you elaborate on the lock file in the <node-manager-lock-directory>?
Does it need to be specified in the broker.xml for both the master and the slave?
Does it actually get used as a “lock” in the concurrency sense, to coordinate the live/backup election?
Thanks
john
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,0rYuECbqGwpo9IjFKJrQQrfyzL5V_aEiqSXV25I7DeF4wuQn3I6z1sJCY1NkNJ-bjdpV8nVix_VDU0mK5tgrxuSQaSyrlVLxE5EiTf0Z8G4F&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Monday, January 15, 2024 9:56 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
In the replication case you still need a PV in K8s, but I don't believe you need to put the whole journal on the PV (and suffer the performance penalty). You should just need to point the node-manager-lock-directory in broker.xml to the PV.
Justin
On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
Based on your reply in the provided link, are you saying that even for the replication HA case, you still need an external FileShare PV, in K8s, so that the nodeID can be persisted between restarts. In this use case, is the performance of this FileShare less of a concern than it is for the shared-store use case?
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
-----Original Message-----
From: Justin Bertram <jb...@apache.org>>
Sent: Friday, January 12, 2024 4:53 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
I sent that email on December 15. You can find it on the web-based mailing
list interface [1].
Justin
[1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l
On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <pa...@hpe.com>>
wrote:
> Hi Justin,
>
> Forgive me but I don’t remember a journal persistence issue discussion. I
> did an email search too and did not see that discussion either. Would you
> please refresh me on the discussion? I may have inadvertently deleted that?
>
> I do see data being replicated to my production slave/standby instance on
> initial startup of the two broker instances. Failover works as expected.
> But when the primary instance starts backup I do not see the active slave
> replicate to the starting primary and I do not see the active slave restart
> and go to the backup server state.
>
> What is the communication sequence when the primary starts up after it
> quit with the active backup broker? In other words, how is the active
> backup broker notified to start replication to the newly started primary
> broker?
>
> I ran the replicated-failback example again with TRACE level logging
> turned on and I do see the attemptFailback=false message in the
> target/server1/log/artemis.log file at startup, so it must be changed later
> or am I not understanding the purpose of this flag?
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> discoveryInitialWaitTimeout=10000}]
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost, b=null],
> nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost)
>
> From my production slave broker log
>
>
> 024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> Thanks,
> Paul
> From: Justin Bertram <jb...@apache.org>>
> Date: Friday, January 12, 2024 at 3:38 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > Is it possible to use jgroups instead of UDP for the cluster connection
> in the example?
>
> Yes. Check out examples/features/clustered/clustered-jgroups for an example
> configuration. Keep in mind that UDP, JGroups, etc. is only used for
> initial discovery. Once brokers are discovered then normal TCP connections
> are created.
>
> Before I continue looking through your logs, etc. can you confirm that you
> resolved the journal persistence issue I described in my previous email? If
> that's not resolved there's no chance for failback to occur.
>
>
> Justin
>
>
>
> On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>>
> wrote:
>
> > Hi Justin,
> >
> > I tried the example and it works. I turned up logging to get a success
> > path through the code to maybe see where things get de-railed. Is it
> > possible to use jgroups instead of UDP for the cluster connection in the
> > example? That would give me a better success signature.
> >
> > My standby instance has this in the broker.xml which is the same as in
> the
> > example:
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > I am seeing this series of log messages that suggests that failback is
> not
> > enabled. Is the attemptFailback=false the default at this point in
> startup
> > and it is changed later or it there something gone wrong at this point?
> >
> > 2024-01-11 18:24:14,066 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > QuorumManager(server=null) about node up =
> > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> > <mailto:
<mailto:%0b>> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> >
> > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > ClientSessionFactoryImpl received backup update for live/backup pair =
> > TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > / null but it didn't belong to TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,067 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Cluster Connected
> >
> > 2024-01-11 18:24:14,068 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting backup manager
> >
> > 2024-01-11 18:24:14,069 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > backup config ClusterConnectionConfiguration{name='my-cluster',
> address='',
> > connectorName='artemis', clientFailureCheckPeriod=30000,
> > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > clusterNotificationAttempts=2}
> >
> > 2024-01-11 18:24:14,081 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > 2024-01-11 18:24:14,083 TRACE
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> >
> > java.lang.Exception:trace<ftp://java.lang.Exception/trace>
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4 rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying reconnection attempt 0/0
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18> > > <mailto:connectorFactory<mailto:connectorFactory>
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18>
> >
> > and currentConnectorConfig: TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Connector NettyConnector
> > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using native epoll
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Started EPOLL Netty Connector version 4.1.86.Final to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Remote destination:
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.25:61616<ftp://10.44.0.25/61616>
> >
> >
> > Thanks,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>>
> > Date: Wednesday, December 13, 2023 at 3:23 PM
> > To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > ActiveMQ Artemis ships with an example in the
> > examples/features/ha/replicated-failback directory that demonstrates how
> > this works. I ran this example and it worked fine. I also started up the
> > brokers manually and triggered failover and failback. Again, everything
> > worked fine. My only guess is that there's something configured wrong or
> > perhaps there's an environmental issue that's preventing failback from
> > working properly for you. If you could provide clear steps on how to
> > reproduce this issue I could investigate further.
> >
> >
> > Justin
> >
> > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am having an issue with failback to the master broker. I start the
> > > master broker then the slave broker. I see the replication from the
> > master
> > > to the slave happen. I stop the master/active broker and a failover
> > happens
> > > to the slave as expected. When the master is started a failback to it
> is
> > > expected but does not happen. The following is my ha-policy config.
> > Did I
> > > miss something in the ha-policy configuration? We are using version
> > 2.28.0
> > > and java17.
> > >
> > > Thanks,
> > > Paul
> > >
> > >
> > > Master broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <master>
> > >
> > > <check-for-live-server>true</check-for-live-server>
> > >
> > > </master>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > Slave broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > The following are the logs from the master and slave brokers
> > > HA “master” broker log after restart
> > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > Starting embedded web server
> > > 2023-12-08 20:42:57,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 15, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,627 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3
> > > 2023-12-08 20:42:57,657 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2> > > > [ID=47edf16f<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2>
> > [ID=47edf16f>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/10.44.0.18:34508<http://10.44.0.18:34508>]]
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > > activemq-branding plugin
> > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > artemis-plugin plugin
> > > 2023-12-08 20:42:57,766 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 18, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,767 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 20, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > Initialising
> > > hawtio services
> > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > Configuration
> > > will be discovered via system properties
> > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > Hawtio 2.15.0
> > > 2023-12-08 20:42:57,975 INFO
> > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > authentication filter, JAAS realm: "activemq" authorized role(s):
> "admin"
> > > role principal classes:
> > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > servlet is disabled
> > > 2023-12-08 20:42:57,989 INFO
> > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > > property: [key=policyLocation,
> > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > HTTP Server started at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > >
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > Artemis Jolokia REST API available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > >
> > >
> > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > Console
> > > available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> > >
> > > 20:42:58,455 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl]
> > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d> > > > [ID=f85ec51b<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d>
> > [ID=f85ec51b>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:49865<http://127.0.0.6:49865>]]
> > > 2023-12-08 20:42:59,524 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858> > > > [ID=7cef23a2<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858>
> > [ID=7cef23a2>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:59691<http://127.0.0.6:59691>]]
> > > 2023-12-08 20:42:59,535 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 22, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,536 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,549 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 24, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,550 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,583 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: OK
> > > 2023-12-08 20:42:59,584 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@5f46053b
> > > 2023-12-08 20:42:59,593 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > 2023-12-08 20:42:59,594 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 28, result: OK
> > > 2023-12-08 20:42:59,616 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:59,615 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 31, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 32, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 35, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 39, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:43:08,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256> > > > [ID=1d1d79ce<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256>
> > [ID=1d1d79ce>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:60787<http://127.0.0.6:60787>]]
> > > 2023-12-08 20:43:08,464 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7> > > > [ID=8f88c6d5<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7>
> > [ID=8f88c6d5>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:40065<http://127.0.0.6:40065>]]
> > >
> > >
> > > HA “slave” broker log.
> > >
> > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > synchronized with live server,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > >
> > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221031: backup announced
> > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221084: Requested 0 quorum votes
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221071: Failing over based on quorum vote results.
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:51,946 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:56,977 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > destroyed
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2023-12-08 20:42:57,445 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6
> > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > > Retrying
> > > 2023-12-08 20:42:57,629 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > > [ID=483170f6<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:55911<http://127.0.0.6:55911>]]
> > > 2023-12-08 20:42:57,630 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > > [ID=4514a2b3<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:60199<http://127.0.0.6:60199>]]
> > > 2023-12-08 20:42:57,648 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > > [ID=fcde4d78<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:53221<http://127.0.0.6:53221>]]
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 29, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 31, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,708 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 33, result: OK
> > > 2023-12-08 20:42:57,709 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@3645037f
> > > 2023-12-08 20:42:57,719 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 35, result: OK
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:57,753 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > 2023-12-08 20:42:57,760 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 38, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, > > > with transaction: null
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 41, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 43, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 44, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 46, result: OK
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 48, result: OK
> > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > >
> > >
> > >
> >
>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by "Shields, Paul Michael" <pa...@hpe.com>.
Hi Justin,
1. To both. The broker runs in a pod and the application server runs in a different pod. There are many application pods (for horizontal scaling). There are two broker pods (primary and standby).
2. Yes, we will have both kinds of pods on a single node. We have a Kubernetes service cluster which the pods run on. We scale the number of worker nodes in the k8s cluster to the system size/load. Each worker node has an application server pod. We have a minimum cluster size of 4 worker nodes. The broker pods have an anti-affinity rule that they can’t run on the same worker node. So in our minimal worker configuration of 4 nodes, we will have an application server pod on each worker node, the primary broker pod on one of the 4 worker nodes, and the standby broker pod on one of the other worker nodes, but not the same node as the primary broker pod is running on.
3. In our application we load balance the application clients across the application servers evenly. We use the MQTT for its “classic” use case for retained messages and last will messages for application server health. The application servers are MQTT publishers, and the application clients are MQTT subscribers. Retained messages are used for new client applications so they have a view of application server state. Last will messages are used to notify the client applications when an application server is no longer responding (“down”). The application clients subscribe to application server health messages (up or down). If an application client received a down message for the application server that it is currently using, then it performs a client side failover to another application server. When the MQTT portion of the client or server application gets disconnected from the broker (broker crash and possible broker failover) it attempts to reconnect to the broker. Once it reconnects to the active broker it receives the retained messages which include any recorded last will messages.
4. Ideally no. If the broker pod just crashes or is restarted the application server is still alive and will continue communication with the client part of our application. We use the application server “down” message to cleanup transactions and client resources, then restart the transaction with a different application server (client failover). If the broker pod never restarts, then the retained messages and last will feature of the broker keep the application server health/state in sync with the clients. Our problem arises after a broker failure and subsequent broker failover, then fail-back. If an application server went down during the broker failover, I would have expected that since the primary broker had a registered last will for a connection that it would be also replicated to the standby broker and the last will would be delivered by the standby broker?
Yes, we are using last-will messages for MQTT client sessions from the application server pod such that if those
sessions die then new, retained messages are sent which reflect the current state. And that works great up until a broker failover occurs and one of our application servers dies at the same time as the primary broker. This can happen during system maintenance when we are applying a security fix to the kernel and a worker node rolling reboot is required to install the new kernel. The problem is that the retained message for the downed application server is replicated to the standby broker and its contents is “server blah is up” because when the application server dies the broker was not up to catch that event and deliver the last will for that application server. The actual state of the application server is now inconsistent with the retained message replicated to the standby broker. The last will message is “Server blah is down” which was not triggered. The problem we are trying to solve is how the standby broker will send a last will for a client that was connected to the primary broker when the broker failover started but does not re-connect to the standby broker. Does such a feature exist? Is this what you meant by “ I don't believe yet-to-be-sent last-will messages will ever be on the backup.”? I am assuming once an application re-connects to the standby broker and registers a last will, that it will be delivered if the client disconnects. When the failed application server does finally connect it will publish its state is up message but the clients that were connected to it now have incomplete in-flight transactions that must be cleaned up. Some of that cleanup must occur on the application server side of the transaction. Until that cleanup occurs those affected clients are not available for processing. A typical client to server ratio is 512 client per server.
We fix this condition by doing a broker scale down to no broker instances to remove all application state, then scale back to primary and standby broker instances. The application state is then reconstructed when the application servers and clients reconnect to the MQTT broker. It was suggested by one of our application developers, “ …can we just clear the broke state somehow and let the client reconnections rebuild the broker state after failover/failback?” The client side of our application maintains a list of available application servers. When the client reconnects to the MQTT broker it compares the retained messages it receives to its list of available application servers. If the retained message is a down message, then that application server is removed from the client’s list of available servers. So, a retained message that states a server is up when in fact is not, triggers retry callbacks for the in-flight transactions. Those timeouts and recovery can take a long time. If the client discovers through the received retained messages that its application server is down, it can cancel the transactions and connect to another application server and retry them there, which is comparatively fairly quick.
Thanks,
Paul
From: Justin Bertram <jb...@apache.org>
Date: Friday, February 2, 2024 at 2:34 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
I'm trying to wrap my head around your deployment, and I have a few
questions...
1) Are your client applications connecting to your "application server
pod" or to an ActiveMQ Artemis broker pod or both?
2) It seems like the failure case you're describing is predicated on both
kinds of pods being on the same node. Is that true? Are both kinds of pods
_always_ deployed on the same node?
3) What is "client-side failover" in the context of the MQTT client
implementation you're using? Based on your description it sounds like it's
just reconnecting which is semantically different from "failover" in my
experience.
4) If the broker pod goes down independently of the application server
pod do you still want to ignore retained messages?
Ideally your application should manage its state without any special
configuration of the broker. Have you considered using last-will messages
for MQTT client sessions from the application server pod such that if those
sessions die then new, retained messages are sent which reflect the current
state? It sounds like something like this would be better than trying to
solve the problem with the broker itself.
Retained messages are stored in queues with the prefix "$sys.mqtt.retain."
but those queues are hard-coded to be durable which means their messages
will always be available on the backup unless persistence for the broker is
completely disabled. I don't believe yet-to-be-sent last-will messages will
ever be on the backup.
Justin
On Thu, Feb 1, 2024 at 5:11 PM Shields, Paul Michael <pa...@hpe.com>
wrote:
> Hi Justin,
>
> After some testing, I have come up with some more questions. One of the
> failure use cases that we are trying to protect against is the loss of a
> node in our Kubernetes cluster that is hosting both the Artemis broker pod
> and one of our server application pods. We have clients that load balance
> connect to our server application pods being hosted on different nodes in
> the Kubernetes cluster. Our client applications use client-side failover
> when the application server pod is marked down in the MQTT broker, the
> client connects to a another application server pod. We are using a single
> “active” MQTT broker so every one of our application clients and servers
> has a complete view of our entire system. In my testing of the above use
> case I see that when the standby instance becomes “active” and clients
> connect to the standby broker instance, they receive a retained message
> that has the state of being “up” which is inconsistent with the actual
> state of the server application pod.
>
> The first question is, can Artemis protect against this use case and what
> broker configuration would you recommend to do so.
>
> We have tried to use a single broker without HA and rely on the Kubernetes
> cluster to restart the broker pod when it detects it is down. But the
> startup times are not consistent enough for our application. Most of the
> time issue is in the inconsistent time required to creating the pod in our
> Kubernetes cluster. With a HA pair of broker pods, the failover
> consistently happens in less than 1 min and that is within our application
> tolerance.
>
> Our application can handle building up system state as our clients connect
> to the MQTT broker as in when the system and broker are first brought up.
> But it does not handle inconsistent state very well.
>
> The second question is, how would we configure Artemis MQTT broker to have
> failover but without replicating the retained and last will messages to the
> standby broker instance? In other words, we would like the system to
> behave as it does on startup after a failover occurs that way our
> application can derive a consistent state of the system as it does on
> startup.
>
> Thanks,
> Paul.
>
> From: Justin Bertram <jb...@apache.org>
> Date: Monday, January 22, 2024 at 9:26 AM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> Looking at the code everything seems to be in order. Can you work up a
> test-case to reproduce the issue you're seeing? Slap it on GitHub, and I'll
> take a look.
>
>
> Justin
>
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by Justin Bertram <jb...@apache.org>.
I'm trying to wrap my head around your deployment, and I have a few
questions...
1) Are your client applications connecting to your "application server
pod" or to an ActiveMQ Artemis broker pod or both?
2) It seems like the failure case you're describing is predicated on both
kinds of pods being on the same node. Is that true? Are both kinds of pods
_always_ deployed on the same node?
3) What is "client-side failover" in the context of the MQTT client
implementation you're using? Based on your description it sounds like it's
just reconnecting which is semantically different from "failover" in my
experience.
4) If the broker pod goes down independently of the application server
pod do you still want to ignore retained messages?
Ideally your application should manage its state without any special
configuration of the broker. Have you considered using last-will messages
for MQTT client sessions from the application server pod such that if those
sessions die then new, retained messages are sent which reflect the current
state? It sounds like something like this would be better than trying to
solve the problem with the broker itself.
Retained messages are stored in queues with the prefix "$sys.mqtt.retain."
but those queues are hard-coded to be durable which means their messages
will always be available on the backup unless persistence for the broker is
completely disabled. I don't believe yet-to-be-sent last-will messages will
ever be on the backup.
Justin
On Thu, Feb 1, 2024 at 5:11 PM Shields, Paul Michael <pa...@hpe.com>
wrote:
> Hi Justin,
>
> After some testing, I have come up with some more questions. One of the
> failure use cases that we are trying to protect against is the loss of a
> node in our Kubernetes cluster that is hosting both the Artemis broker pod
> and one of our server application pods. We have clients that load balance
> connect to our server application pods being hosted on different nodes in
> the Kubernetes cluster. Our client applications use client-side failover
> when the application server pod is marked down in the MQTT broker, the
> client connects to a another application server pod. We are using a single
> “active” MQTT broker so every one of our application clients and servers
> has a complete view of our entire system. In my testing of the above use
> case I see that when the standby instance becomes “active” and clients
> connect to the standby broker instance, they receive a retained message
> that has the state of being “up” which is inconsistent with the actual
> state of the server application pod.
>
> The first question is, can Artemis protect against this use case and what
> broker configuration would you recommend to do so.
>
> We have tried to use a single broker without HA and rely on the Kubernetes
> cluster to restart the broker pod when it detects it is down. But the
> startup times are not consistent enough for our application. Most of the
> time issue is in the inconsistent time required to creating the pod in our
> Kubernetes cluster. With a HA pair of broker pods, the failover
> consistently happens in less than 1 min and that is within our application
> tolerance.
>
> Our application can handle building up system state as our clients connect
> to the MQTT broker as in when the system and broker are first brought up.
> But it does not handle inconsistent state very well.
>
> The second question is, how would we configure Artemis MQTT broker to have
> failover but without replicating the retained and last will messages to the
> standby broker instance? In other words, we would like the system to
> behave as it does on startup after a failover occurs that way our
> application can derive a consistent state of the system as it does on
> startup.
>
> Thanks,
> Paul.
>
> From: Justin Bertram <jb...@apache.org>
> Date: Monday, January 22, 2024 at 9:26 AM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> Looking at the code everything seems to be in order. Can you work up a
> test-case to reproduce the issue you're seeing? Slap it on GitHub, and I'll
> take a look.
>
>
> Justin
>
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by "Shields, Paul Michael" <pa...@hpe.com>.
Hi Justin,
After some testing, I have come up with some more questions. One of the failure use cases that we are trying to protect against is the loss of a node in our Kubernetes cluster that is hosting both the Artemis broker pod and one of our server application pods. We have clients that load balance connect to our server application pods being hosted on different nodes in the Kubernetes cluster. Our client applications use client-side failover when the application server pod is marked down in the MQTT broker, the client connects to a another application server pod. We are using a single “active” MQTT broker so every one of our application clients and servers has a complete view of our entire system. In my testing of the above use case I see that when the standby instance becomes “active” and clients connect to the standby broker instance, they receive a retained message that has the state of being “up” which is inconsistent with the actual state of the server application pod.
The first question is, can Artemis protect against this use case and what broker configuration would you recommend to do so.
We have tried to use a single broker without HA and rely on the Kubernetes cluster to restart the broker pod when it detects it is down. But the startup times are not consistent enough for our application. Most of the time issue is in the inconsistent time required to creating the pod in our Kubernetes cluster. With a HA pair of broker pods, the failover consistently happens in less than 1 min and that is within our application tolerance.
Our application can handle building up system state as our clients connect to the MQTT broker as in when the system and broker are first brought up. But it does not handle inconsistent state very well.
The second question is, how would we configure Artemis MQTT broker to have failover but without replicating the retained and last will messages to the standby broker instance? In other words, we would like the system to behave as it does on startup after a failover occurs that way our application can derive a consistent state of the system as it does on startup.
Thanks,
Paul.
From: Justin Bertram <jb...@apache.org>
Date: Monday, January 22, 2024 at 9:26 AM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
Looking at the code everything seems to be in order. Can you work up a
test-case to reproduce the issue you're seeing? Slap it on GitHub, and I'll
take a look.
Justin
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by "Shields, Paul Michael" <pa...@hpe.com>.
I’ll put together a test case.
Thanks,
Paul
From: Justin Bertram <jb...@apache.org>
Date: Monday, January 22, 2024 at 9:26 AM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
Looking at the code everything seems to be in order. Can you work up a
test-case to reproduce the issue you're seeing? Slap it on GitHub, and I'll
take a look.
Justin
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by Justin Bertram <jb...@apache.org>.
Looking at the code everything seems to be in order. Can you work up a
test-case to reproduce the issue you're seeing? Slap it on GitHub, and I'll
take a look.
Justin
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by "Shields, Paul Michael" <pa...@hpe.com>.
We are using QoS = 1.
Paul
From: Justin Bertram <jb...@apache.org>
Date: Monday, January 22, 2024 at 9:09 AM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
What QoS are you using to send the message(s)? Messages sent with QoS 0 are
non-durable and > 0 are durable.
Justin
On Mon, Jan 22, 2024 at 8:51 AM Shields, Paul Michael <pa...@hpe.com>
wrote:
> Hi Justin,
>
> A follow-on question about persistent data and failover. The
> documentation makes this statement. “When using replication, the live and
> the backup servers do not share the same data directories, all data
> synchronization is done over the network. Therefore all (persistent) data
> received by the live server will be duplicated to the backup.” [1]. We are
> using the MQTT protocol. We have published messages that have the retained
> flag set. We also use “last will” messages and set the lastWillRetain flag
> to true. When the last will message is sent by the broker the retained
> message for the publisher is updated with the last will message. Now for
> the question. Are last will messages considered persistent data and are
> duplicated to the backup or is the retained message updated with the last
> will message duplicated to the backup broker? We seem to be losing last
> will messages on a broker failover (ie the retained message does not
> contain the last will message).
>
> Thanks,
> Paul
>
> [1]:
> https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies>
>
> From: Justin Bertram <jb...@apache.org>
> Date: Friday, January 19, 2024 at 11:47 AM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > So even though the persistent data will be stored pod-local...
>
> If you disable persistence on the broker then no data will be persisted on
> the local pod.
>
> > ...enabling persistence has the knock-on effect of also making it
> available to the backup server in addition to whatever the “replication”
> model does?
>
> Only persistent data is replicated to the backup. This is noted in the
> documentation [1].
>
> To be clear, this is not "in addition" to what replication is doing. The
> replication manager "listens" for persisted data and that's the information
> that it sends to the backup. If you disable persistence then no data is
> replicated.
>
> Generally speaking, client applications make a basic decision about the
> messages they send - persistent or non-persistent. Sending persistent
> messages comes with certain reliability guarantees but worse performance.
> Sending non-persistent messages comes with better performance but less
> reliability. The ability to survive a failure is one of the reliability
> benefits enjoyed by persistent messages.
>
>
> Justin
>
> [1]
> https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies>
> <
> https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies>
> >
>
> On Fri, Jan 19, 2024 at 10:50 AM John Lilley
> <jo...@redpointglobal.com.invalid> wrote:
> Hi Justin,
>
> This does seem counter-intuitive. We’re using replication, not shared
> storage. So even though the persistent data will be stored pod-local,
> enabling persistence has the knock-on effect of also making it available to
> the backup server in addition to whatever the “replication” model does?
>
> john
>
>
>
> [rg]<https://www.redpointglobal.com/ >
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com
> <ma...@redpointglobal.com>
> From: Justin Bertram <jb...@apache.org>>
> Sent: Friday, January 19, 2024 9:42 AM
> To: users@activemq.apache.org<ma...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
>
> *** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ***
>
> > ...do we currently set persistenceEnable=false? I would expect there’s
> no reason for setting it true, since the pod-local storage will evaporate
> with the pod.
>
> You definitely want to enable persistence because only persistent data
> will be available on the backup.
>
>
> Justin
>
> On Fri, Jan 19, 2024 at 10:21 AM John Lilley
> <john.lilley@redpointglobal.com.invalid<mailto:
> john.lilley@redpointglobal.com.invalid>> wrote:
> Hi Paul,
>
> Thanks for the information! It sounds like you have experience with AMQ
> in K8S. Can you share your two broker.xml files with us?
>
> Lino: do we currently set persistenceEnable=false? I would expect
> there’s no reason for setting it true, since the pod-local storage will
> evaporate with the pod.
>
> Thanks
> John
>
>
>
>
> Error! Filename not specified.<
> https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
> >
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com
> <ma...@redpointglobal.com>
> From: Shields, Paul Michael <paul.shields@hpe.com<mailto:
> paul.shields@hpe.com>>
> Sent: Friday, January 19, 2024 8:51 AM
> To: users@activemq.apache.org<ma...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
>
> *** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ***
>
> Hi John,
>
> The <node-manager-lock-directory>. Needs to be specified in both the
> master and slave broker.xml files.
>
> the nodeUUID which is contained in the server lock file was changing
> between active broker startups. The server.lock file is stored in the
> /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our
> current configuration of the broker has persistenceEnable=false which
> causes any data/broker state to be removed when the broker pod is
> destroyed. Thus a new nodeUUID was being generated on each Primary broker
> startup. When the Primary broker instance starts up it looks for an
> existing server.lock file. When it does not find one it generates a new
> nodeUUID and places it in the server.lock file, otherwise it uses the
> nodeUUID contained within the existing server.lock file. The Primary broker
> then communicates this nodeUUID through the cluster connection to the
> Standby broker instance on startup. The Standby broker instance uses the
> nodeUUID to determine if the Primary broker instance is part of its HA
> pair. The Standby broker instance saved off the original nodeUUID when the
> HA cluster was first started and since the recently communicated nodeUUID
> does not match the nodeUUID that it previously stored the fail back is not
> performed. Once persistence was enabled the Standby broker instance
> performed the fallback to the Primary instance and was no longer in the
> "active" state
>
> This is what I have observed in our Kubernetes environment. Justin please
> make any corrections to my description of the nodeUUID use.
>
> Paul
> From: John Lilley <john.lilley@redpointglobal.com.INVALID<mailto:
> john.lilley@redpointglobal.com.INVALID>>
> Date: Thursday, January 18, 2024 at 5:49 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org> <
> users@activemq.apache.org<ma...@activemq.apache.org>>
> Subject: RE: Trouble with Replication HA Master/Slave config performing
> failback
> Hi Justin,
>
> Can you elaborate on the lock file in the <node-manager-lock-directory>?
> Does it need to be specified in the broker.xml for both the master and the
> slave?
> Does it actually get used as a “lock” in the concurrency sense, to
> coordinate the live/backup election?
>
> Thanks
> john
>
>
>
> Error! Filename not specified.<
> https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,0rYuECbqGwpo9IjFKJrQQrfyzL5V_aEiqSXV25I7DeF4wuQn3I6z1sJCY1NkNJ-bjdpV8nVix_VDU0mK5tgrxuSQaSyrlVLxE5EiTf0Z8G4F&typo=1<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,0rYuECbqGwpo9IjFKJrQQrfyzL5V_aEiqSXV25I7DeF4wuQn3I6z1sJCY1NkNJ-bjdpV8nVix_VDU0mK5tgrxuSQaSyrlVLxE5EiTf0Z8G4F&typo=1>
> >
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com
> <ma...@redpointglobal.com>
> From: Justin Bertram <jb...@apache.org>>
> Sent: Monday, January 15, 2024 9:56 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
>
> *** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ***
>
> In the replication case you still need a PV in K8s, but I don't believe
> you need to put the whole journal on the PV (and suffer the performance
> penalty). You should just need to point the node-manager-lock-directory in
> broker.xml to the PV.
>
>
> Justin
>
> On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira
> <lino.pereira@redpointglobal.com.invalid<mailto:
> lino.pereira@redpointglobal.com.invalid>> wrote:
>
> Hi Justin,
>
> Based on your reply in the provided link, are you saying that even for the
> replication HA case, you still need an external FileShare PV, in K8s, so
> that the nodeID can be persisted between restarts. In this use case, is the
> performance of this FileShare less of a concern than it is for the
> shared-store use case?
>
> Thanks,
> Lino
>
>
> Error! Filename not specified.<
> https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1>
> >
>
> Lino Pereira
>
> C++ Developer, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>>
> Sent: Friday, January 12, 2024 4:53 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
>
> *** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ***
>
> I sent that email on December 15. You can find it on the web-based mailing
> list interface [1].
>
> Justin
>
> [1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l<https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l> <
> https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l<https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l> >
>
> On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>>
> wrote:
>
> > Hi Justin,
> >
> > Forgive me but I don’t remember a journal persistence issue discussion. I
> > did an email search too and did not see that discussion either. Would you
> > please refresh me on the discussion? I may have inadvertently deleted
> that?
> >
> > I do see data being replicated to my production slave/standby instance on
> > initial startup of the two broker instances. Failover works as expected.
> > But when the primary instance starts backup I do not see the active slave
> > replicate to the starting primary and I do not see the active slave
> restart
> > and go to the backup server state.
> >
> > What is the communication sequence when the primary starts up after it
> > quit with the active backup broker? In other words, how is the active
> > backup broker notified to start replication to the newly started primary
> > broker?
> >
> > I ran the replicated-failback example again with TRACE level logging
> > turned on and I do see the attemptFailback=false message in the
> > target/server1/log/artemis.log file at startup, so it must be changed
> later
> > or am I not understanding the purpose of this flag?
> >
> > 2024-01-12 14:40:18,341 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> > discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=localhost, b=null],
> > nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=localhost)
> >
> > From my production slave broker log
> >
> >
> > 024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > Thanks,
> > Paul
> > From: Justin Bertram <jb...@apache.org>>
> > Date: Friday, January 12, 2024 at 3:38 PM
> > To: users@activemq.apache.org<ma...@activemq.apache.org> <
> users@activemq.apache.org<ma...@activemq.apache.org>>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > > Is it possible to use jgroups instead of UDP for the cluster connection
> > in the example?
> >
> > Yes. Check out examples/features/clustered/clustered-jgroups for an
> example
> > configuration. Keep in mind that UDP, JGroups, etc. is only used for
> > initial discovery. Once brokers are discovered then normal TCP
> connections
> > are created.
> >
> > Before I continue looking through your logs, etc. can you confirm that
> you
> > resolved the journal persistence issue I described in my previous email?
> If
> > that's not resolved there's no chance for failback to occur.
> >
> >
> > Justin
> >
> >
> >
> > On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> > paul.shields@hpe.com<ma...@hpe.com>>
> > wrote:
> >
> > > Hi Justin,
> > >
> > > I tried the example and it works. I turned up logging to get a success
> > > path through the code to maybe see where things get de-railed. Is it
> > > possible to use jgroups instead of UDP for the cluster connection in
> the
> > > example? That would give me a better success signature.
> > >
> > > My standby instance has this in the broker.xml which is the same as in
> > the
> > > example:
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > I am seeing this series of log messages that suggests that failback is
> > not
> > > enabled. Is the attemptFailback=false the default at this point in
> > startup
> > > and it is changed later or it there something gone wrong at this point?
> > >
> > > 2024-01-11 18:24:14,066 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.Topology]
> Topology@7e0a913c
> > [owner=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > >
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > > QuorumManager(server=null) about node up =
> > > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > > Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null]
> > >
> > > 2024-01-11 18:24:14,067 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.Topology]
> Topology@7e0a913c
> > [owner=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > >
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > >
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> <mailto:
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> >
> > > <mailto:
> <mailto:%0b>> >
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> <mailto:
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> >
> > >
> > > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > > Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null]
> > >
> > > 2024-01-11 18:24:14,067 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > ClientSessionFactoryImpl received backup update for live/backup pair =
> > > TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > > / null but it didn't belong to TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > >
> > > 2024-01-11 18:24:14,067 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Cluster Connected
> > >
> > > 2024-01-11 18:24:14,068 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Starting backup manager
> > >
> > > 2024-01-11 18:24:14,069 DEBUG
> > > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > > backup config ClusterConnectionConfiguration{name='my-cluster',
> > address='',
> > > connectorName='artemis', clientFailureCheckPeriod=30000,
> > > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > > maxRetryInterval=32000, initialConnectAttempts=20,
> reconnectAttempts=-1,
> > > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > > staticConnectors=[], discoveryGroupName='my-discovery-group',
> maxHops=1,
> > > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > > clusterNotificationAttempts=2}
> > >
> > > 2024-01-11 18:24:14,081 DEBUG
> > > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > > BackupManager connecting to DiscoveryBackupConnector
> > > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> > >
> > > 2024-01-11 18:24:14,082 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Starting Backup Server
> > >
> > > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Setting server state as started
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > looking up the node through nodeLocator.locateNode()
> > >
> > > 2024-01-11 18:24:14,083 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Connecting towards a possible live, connection
> > > information=Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> > >
> > > 2024-01-11 18:24:14,083 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Calling
> > >
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> > >
> > > java.lang.Exception:trace<ftp://java.lang.Exception/trace >
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(
> https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>
> <
> https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1> >)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0<
> ftp://artemis-core-client-2.28.0.jar/2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0> >]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(
> https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>
> <
> https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1> >)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0<
> ftp://artemis-core-client-2.28.0.jar/2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0> >]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(
> https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>
> <
> https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1> >)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0<
> ftp://artemis-core-client-2.28.0.jar/2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0> >]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(
> https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>
> <
> https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1> >)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0<
> INVALID URI REMOVED<INVALID URI REMOVED>t-2.28.0.jar/2.28.0__;!!NpxR!id1NAxyJS3ZvxYyFxL1hW6kie0R0DxtHcpGwZ2JPCSiaWM-kAE0o-eqMalz3uqBqHxLBPLNhV_h_Cp3uPg$ >]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(
> https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>
> <
> https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1> >)
> > > ~[artemis-server-2.28.0.jar:2.28.0<
> ftp://artemis-server-2.28.0.jar/2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0> >]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(
> https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>
> <
> https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1> >)
> > > ~[artemis-server-2.28.0.jar:2.28.0<
> ftp://artemis-server-2.28.0.jar/2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0> >]
> > >
> > > at
> > >
> >
> https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>
> <
> https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>
> >(
> https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4>
> rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<
> https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1> >)
> > > ~[artemis-server-2.28.0.jar:2.28.0<
> ftp://artemis-server-2.28.0.jar/2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0> >]
> > >
> > > at
> > >
> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$
> https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>
> <
> https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>
> >(
> https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>
> <
> https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1> >)
> > > ~[artemis-server-2.28.0.jar:2.28.0<
> ftp://artemis-server-2.28.0.jar/2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0> >]
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > Trying reconnection attempt 0/0
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > Trying to connect with
> > >
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> <mailto:connectorFactory
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18>
> > > <mailto:connectorFactory<mailto:connectorFactory<mailto:connectorFactory%3cmailto:connectorFactory>>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> <mailto:
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> >
> > >
> > > and currentConnectorConfig: TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Connector NettyConnector
> > > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> > useServlet=false,
> > > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > > using native epoll
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Started EPOLL Netty Connector version 4.1.86.Final to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<
> ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Remote destination:
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.25:61616<ftp://10.44.0.25/61616 >
> > >
> > >
> > > Thanks,
> > > Paul
> > >
> > > From: Justin Bertram <jb...@apache.org>>
> > > Date: Wednesday, December 13, 2023 at 3:23 PM
> > > To: users@activemq.apache.org<ma...@activemq.apache.org> <
> users@activemq.apache.org<ma...@activemq.apache.org>>
> > > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > > failback
> > > ActiveMQ Artemis ships with an example in the
> > > examples/features/ha/replicated-failback directory that demonstrates
> how
> > > this works. I ran this example and it worked fine. I also started up
> the
> > > brokers manually and triggered failover and failback. Again, everything
> > > worked fine. My only guess is that there's something configured wrong
> or
> > > perhaps there's an environmental issue that's preventing failback from
> > > working properly for you. If you could provide clear steps on how to
> > > reproduce this issue I could investigate further.
> > >
> > >
> > > Justin
> > >
> > > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> > paul.shields@hpe.com<ma...@hpe.com>
> > > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I am having an issue with failback to the master broker. I start the
> > > > master broker then the slave broker. I see the replication from the
> > > master
> > > > to the slave happen. I stop the master/active broker and a failover
> > > happens
> > > > to the slave as expected. When the master is started a failback to it
> > is
> > > > expected but does not happen. The following is my ha-policy config.
> > > Did I
> > > > miss something in the ha-policy configuration? We are using version
> > > 2.28.0
> > > > and java17.
> > > >
> > > > Thanks,
> > > > Paul
> > > >
> > > >
> > > > Master broker.xml config
> > > >
> > > > <ha-policy>
> > > >
> > > > <replication>
> > > >
> > > > <master>
> > > >
> > > > <check-for-live-server>true</check-for-live-server>
> > > >
> > > > </master>
> > > >
> > > > </replication>
> > > >
> > > > </ha-policy>
> > > >
> > > > Slave broker.xml config
> > > >
> > > > <ha-policy>
> > > >
> > > > <replication>
> > > >
> > > > <slave>
> > > >
> > > > <allow-failback>true</allow-failback>
> > > >
> > > > </slave>
> > > >
> > > > </replication>
> > > >
> > > > </ha-policy>
> > > >
> > > > The following are the logs from the master and slave brokers
> > > > HA “master” broker log after restart
> > > > 2023-12-08 20:42:57,509 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<
> ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> >
> > for
> > > > protocols [MQTT]
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > >
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<
> ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > for
> > > > protocols [CORE]
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221007: Server is now live
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > > Starting embedded web server
> > > > 2023-12-08 20:42:57,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 15, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,627 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3
> > > > 2023-12-08 20:42:57,657 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> <mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2>
> > > > [ID=47edf16f<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> <mailto:
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> >
> > > [ID=47edf16f>,
> > > > local= /10.44.0.18:61616<http://10.44.0.18:61616 >, remote=/
> 10.44.0.18:34508<http://10.44.0.18:34508 >]]
> > > > 2023-12-08 20:42:57,720 INFO
> > > > [org.apache.activemq.hawtio.branding.PluginContextListener]
> Initialized
> > > > activemq-branding plugin
> > > > 2023-12-08 20:42:57,737 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is connected
> > > > 2023-12-08 20:42:57,755 INFO
> > > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > > artemis-plugin plugin
> > > > 2023-12-08 20:42:57,766 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 18, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,767 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 20, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > > Initialising
> > > > hawtio services
> > > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > > Configuration
> > > > will be discovered via system properties
> > > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > > Hawtio 2.15.0
> > > > 2023-12-08 20:42:57,975 INFO
> > > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > > authentication filter, JAAS realm: "activemq" authorized role(s):
> > "admin"
> > > > role principal classes:
> > > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > > servlet is disabled
> > > > 2023-12-08 20:42:57,989 INFO
> > > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia
> overridden
> > > > property: [key=policyLocation,
> > > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > > HTTP Server started at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> >
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> >
> > >
> > > >
> > > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > > Artemis Jolokia REST API available at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> >
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> >
> > >
> > > >
> > > > 2023-12-08
> > > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08> >
> > >
> > > >
> > > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > > Console
> > > > available at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> >
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> >
> > >
> > > >
> > > > 2023-12-08
> > > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08>
> >
> > >
> > > >
> > > > 20:42:58,455 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl]
> > > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> <mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d>
> > > > [ID=f85ec51b<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> <mailto:
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> >
> > > [ID=f85ec51b>,
> > > > local= /10.44.0.18:61616<http://10.44.0.18:61616 >, remote=/
> 127.0.0.6:49865<http://127.0.0.6:49865 >]]
> > > > 2023-12-08 20:42:59,524 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> <mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858>
> > > > [ID=7cef23a2<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> <mailto:
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> >
> > > [ID=7cef23a2>,
> > > > local= /10.44.0.18:61616<http://10.44.0.18:61616 >, remote=/
> 127.0.0.6:59691<http://127.0.0.6:59691 >]]
> > > > 2023-12-08 20:42:59,535 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 22, result: NO_BINDINGS
> > > > 2023-12-08 20:42:59,536 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> > connectionID:
> > > > 7cef23a2
> > > > 2023-12-08 20:42:59,549 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 24, result: NO_BINDINGS
> > > > 2023-12-08 20:42:59,550 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> > connectionID:
> > > > 7cef23a2
> > > > 2023-12-08 20:42:59,583 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 26, result: OK
> > > > 2023-12-08 20:42:59,584 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=true]@5f46053b
> > > > 2023-12-08 20:42:59,593 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> > created
> > > > consumer with ID: 0, with session name:
> > > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > > 2023-12-08 20:42:59,594 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 28, result: OK
> > > > 2023-12-08 20:42:59,616 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > > 2023-12-08 20:42:59,615 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085,
> > > > with transaction: null
> > > > 2023-12-08 20:42:59,621 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 31, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,621 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
> > > > with transaction: null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 32, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
> > > > with transaction: null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 35, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293,
> > > > with transaction: null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 39, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:43:08,447 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> <mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256>
> > > > [ID=1d1d79ce<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> <mailto:
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> >
> > > [ID=1d1d79ce>,
> > > > local= /10.44.0.18:61616<http://10.44.0.18:61616 >, remote=/
> 127.0.0.6:60787<http://127.0.0.6:60787 >]]
> > > > 2023-12-08 20:43:08,464 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> <mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7>
> > > > [ID=8f88c6d5<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> <mailto:
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> >
> > > [ID=8f88c6d5>,
> > > > local= /10.44.0.18:61616<http://10.44.0.18:61616 >, remote=/
> 127.0.0.6:40065<http://127.0.0.6:40065 >]]
> > > >
> > > >
> > > > HA “slave” broker log.
> > > >
> > > > 2023-12-08 20:37:58,374 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0
> [null]
> > > > started, waiting live to fail before it gets active
> > > >
> > > > 2023-12-08 20:37:59,020 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > > synchronized with live server,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > > >
> > > > 2023-12-08 20:38:00,032 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221031: backup announced
> > > > 2023-12-08 20:42:27,415 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221084: Requested 0 quorum votes
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221071: Failing over based on quorum vote results.
> > > > 2023-12-08 20:42:27,431 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212037: Connection failure to
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > > 10.44.0.20:61616<ftp://10.44.0.20/61616 > has been detected:
> AMQ219015: The connection was
> > > > disconnected because of server shutdown [code=DISCONNECTED]
> > > > 2023-12-08 20:42:27,431 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212037: Connection failure to
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > > 10.44.0.20:61616<ftp://10.44.0.20/61616 > has been detected:
> AMQ219015: The connection was
> > > > disconnected because of server shutdown [code=DISCONNECTED]
> > > > 2023-12-08 20:42:27,434 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > > 2023-12-08 20:42:27,574 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > > 2023-12-08 20:42:27,575 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > > 2023-12-08 20:42:27,578 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > > 2023-12-08 20:42:27,578 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > > 2023-12-08 20:42:27,611 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221007: Server is now live
> > > > 2023-12-08 20:42:27,699 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<
> ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> >
> > for
> > > > protocols [MQTT]
> > > > 2023-12-08 20:42:27,717 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > >
> cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<
> ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > for
> > > > protocols [CORE]
> > > > 2023-12-08 20:42:51,946 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > [ID=df357d0e>,
> > > > local= /10.40.0.106:61616<http://10.40.0.106:61616 >, remote=/
> 127.0.0.6:56167<http://127.0.0.6:56167 >]]
> > > > 2023-12-08 20:42:56,977 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > > destroyed
> > > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > [ID=df357d0e>,
> > > > local= /10.40.0.106:61616<http://10.40.0.106:61616 >, remote=/
> 127.0.0.6:56167<http://127.0.0.6:56167 >]]
> > > > 2023-12-08 20:42:57,443 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212004: Failed to connect to server.
> > > > 2023-12-08 20:42:57,445 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 26, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,447 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6
> > > > 2023-12-08 20:42:57,512 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > > failure detected. Unblocking a blocking call that will never get a
> > > response
> > > > 2023-12-08 20:42:57,513 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is unable to connect to
> destination.
> > > > Retrying
> > > > 2023-12-08 20:42:57,629 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > > > [ID=483170f6<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > > [ID=483170f6>,
> > > > local= /10.40.0.106:61616<http://10.40.0.106:61616 >, remote=/
> 127.0.0.6:55911<http://127.0.0.6:55911 >]]
> > > > 2023-12-08 20:42:57,630 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > > > [ID=4514a2b3<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > > [ID=4514a2b3>,
> > > > local= /10.40.0.106:61616<http://10.40.0.106:61616 >, remote=/
> 127.0.0.6:60199<http://127.0.0.6:60199 >]]
> > > > 2023-12-08 20:42:57,648 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > > > [ID=fcde4d78<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > > [ID=fcde4d78>,
> > > > local= /10.40.0.106:61616<http://10.40.0.106:61616 >, remote=/
> 127.0.0.6:53221<http://127.0.0.6:53221 >]]
> > > > 2023-12-08 20:42:57,663 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 29, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,663 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> > connectionID:
> > > > fcde4d78
> > > > 2023-12-08 20:42:57,679 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 31, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,679 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> > connectionID:
> > > > fcde4d78
> > > > 2023-12-08 20:42:57,708 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 33, result: OK
> > > > 2023-12-08 20:42:57,709 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=true]@3645037f
> > > > 2023-12-08 20:42:57,719 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> > created
> > > > consumer with ID: 0, with session name:
> > > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > > 2023-12-08 20:42:57,720 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 35, result: OK
> > > > 2023-12-08 20:42:57,755 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > > 2023-12-08 20:42:57,753 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563,
> > > > with transaction: null
> > > > 2023-12-08 20:42:57,760 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 38, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,761 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
> > > > with transaction: null
> > > > 2023-12-08 20:42:57,761 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 41, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
> > > > with transaction: null
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 43, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555,
> > > > with transaction: null
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 44, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:59,607 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is connected
> > > > 2023-12-08 20:42:59,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 46, result: OK
> > > > 2023-12-08 20:42:59,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 48, result: OK
> > > > 2023-12-08 20:43:07,437 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > > failure detected. Unblocking a blocking call that will never get a
> > > response
> > > >
> > > >
> > > >
> > >
> >
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by Justin Bertram <jb...@apache.org>.
What QoS are you using to send the message(s)? Messages sent with QoS 0 are
non-durable and > 0 are durable.
Justin
On Mon, Jan 22, 2024 at 8:51 AM Shields, Paul Michael <pa...@hpe.com>
wrote:
> Hi Justin,
>
> A follow-on question about persistent data and failover. The
> documentation makes this statement. “When using replication, the live and
> the backup servers do not share the same data directories, all data
> synchronization is done over the network. Therefore all (persistent) data
> received by the live server will be duplicated to the backup.” [1]. We are
> using the MQTT protocol. We have published messages that have the retained
> flag set. We also use “last will” messages and set the lastWillRetain flag
> to true. When the last will message is sent by the broker the retained
> message for the publisher is updated with the last will message. Now for
> the question. Are last will messages considered persistent data and are
> duplicated to the backup or is the retained message updated with the last
> will message duplicated to the backup broker? We seem to be losing last
> will messages on a broker failover (ie the retained message does not
> contain the last will message).
>
> Thanks,
> Paul
>
> [1]:
> https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies
>
> From: Justin Bertram <jb...@apache.org>
> Date: Friday, January 19, 2024 at 11:47 AM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > So even though the persistent data will be stored pod-local...
>
> If you disable persistence on the broker then no data will be persisted on
> the local pod.
>
> > ...enabling persistence has the knock-on effect of also making it
> available to the backup server in addition to whatever the “replication”
> model does?
>
> Only persistent data is replicated to the backup. This is noted in the
> documentation [1].
>
> To be clear, this is not "in addition" to what replication is doing. The
> replication manager "listens" for persisted data and that's the information
> that it sends to the backup. If you disable persistence then no data is
> replicated.
>
> Generally speaking, client applications make a basic decision about the
> messages they send - persistent or non-persistent. Sending persistent
> messages comes with certain reliability guarantees but worse performance.
> Sending non-persistent messages comes with better performance but less
> reliability. The ability to survive a failure is one of the reliability
> benefits enjoyed by persistent messages.
>
>
> Justin
>
> [1]
> https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies
> <
> https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies
> >
>
> On Fri, Jan 19, 2024 at 10:50 AM John Lilley
> <jo...@redpointglobal.com.invalid> wrote:
> Hi Justin,
>
> This does seem counter-intuitive. We’re using replication, not shared
> storage. So even though the persistent data will be stored pod-local,
> enabling persistence has the knock-on effect of also making it available to
> the backup server in addition to whatever the “replication” model does?
>
> john
>
>
>
> [rg]<https://www.redpointglobal.com/>
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com
> <ma...@redpointglobal.com>
> From: Justin Bertram <jb...@apache.org>>
> Sent: Friday, January 19, 2024 9:42 AM
> To: users@activemq.apache.org<ma...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
>
> *** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ***
>
> > ...do we currently set persistenceEnable=false? I would expect there’s
> no reason for setting it true, since the pod-local storage will evaporate
> with the pod.
>
> You definitely want to enable persistence because only persistent data
> will be available on the backup.
>
>
> Justin
>
> On Fri, Jan 19, 2024 at 10:21 AM John Lilley
> <john.lilley@redpointglobal.com.invalid<mailto:
> john.lilley@redpointglobal.com.invalid>> wrote:
> Hi Paul,
>
> Thanks for the information! It sounds like you have experience with AMQ
> in K8S. Can you share your two broker.xml files with us?
>
> Lino: do we currently set persistenceEnable=false? I would expect
> there’s no reason for setting it true, since the pod-local storage will
> evaporate with the pod.
>
> Thanks
> John
>
>
>
>
> Error! Filename not specified.<
> https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1
> >
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com
> <ma...@redpointglobal.com>
> From: Shields, Paul Michael <paul.shields@hpe.com<mailto:
> paul.shields@hpe.com>>
> Sent: Friday, January 19, 2024 8:51 AM
> To: users@activemq.apache.org<ma...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
>
> *** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ***
>
> Hi John,
>
> The <node-manager-lock-directory>. Needs to be specified in both the
> master and slave broker.xml files.
>
> the nodeUUID which is contained in the server lock file was changing
> between active broker startups. The server.lock file is stored in the
> /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our
> current configuration of the broker has persistenceEnable=false which
> causes any data/broker state to be removed when the broker pod is
> destroyed. Thus a new nodeUUID was being generated on each Primary broker
> startup. When the Primary broker instance starts up it looks for an
> existing server.lock file. When it does not find one it generates a new
> nodeUUID and places it in the server.lock file, otherwise it uses the
> nodeUUID contained within the existing server.lock file. The Primary broker
> then communicates this nodeUUID through the cluster connection to the
> Standby broker instance on startup. The Standby broker instance uses the
> nodeUUID to determine if the Primary broker instance is part of its HA
> pair. The Standby broker instance saved off the original nodeUUID when the
> HA cluster was first started and since the recently communicated nodeUUID
> does not match the nodeUUID that it previously stored the fail back is not
> performed. Once persistence was enabled the Standby broker instance
> performed the fallback to the Primary instance and was no longer in the
> "active" state
>
> This is what I have observed in our Kubernetes environment. Justin please
> make any corrections to my description of the nodeUUID use.
>
> Paul
> From: John Lilley <john.lilley@redpointglobal.com.INVALID<mailto:
> john.lilley@redpointglobal.com.INVALID>>
> Date: Thursday, January 18, 2024 at 5:49 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org> <
> users@activemq.apache.org<ma...@activemq.apache.org>>
> Subject: RE: Trouble with Replication HA Master/Slave config performing
> failback
> Hi Justin,
>
> Can you elaborate on the lock file in the <node-manager-lock-directory>?
> Does it need to be specified in the broker.xml for both the master and the
> slave?
> Does it actually get used as a “lock” in the concurrency sense, to
> coordinate the live/backup election?
>
> Thanks
> john
>
>
>
> Error! Filename not specified.<
> https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,0rYuECbqGwpo9IjFKJrQQrfyzL5V_aEiqSXV25I7DeF4wuQn3I6z1sJCY1NkNJ-bjdpV8nVix_VDU0mK5tgrxuSQaSyrlVLxE5EiTf0Z8G4F&typo=1
> >
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com
> <ma...@redpointglobal.com>
> From: Justin Bertram <jb...@apache.org>>
> Sent: Monday, January 15, 2024 9:56 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
>
> *** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ***
>
> In the replication case you still need a PV in K8s, but I don't believe
> you need to put the whole journal on the PV (and suffer the performance
> penalty). You should just need to point the node-manager-lock-directory in
> broker.xml to the PV.
>
>
> Justin
>
> On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira
> <lino.pereira@redpointglobal.com.invalid<mailto:
> lino.pereira@redpointglobal.com.invalid>> wrote:
>
> Hi Justin,
>
> Based on your reply in the provided link, are you saying that even for the
> replication HA case, you still need an external FileShare PV, in K8s, so
> that the nodeID can be persisted between restarts. In this use case, is the
> performance of this FileShare less of a concern than it is for the
> shared-store use case?
>
> Thanks,
> Lino
>
>
> Error! Filename not specified.<
> https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1
> >
>
> Lino Pereira
>
> C++ Developer, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>>
> Sent: Friday, January 12, 2024 4:53 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
>
> *** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ***
>
> I sent that email on December 15. You can find it on the web-based mailing
> list interface [1].
>
> Justin
>
> [1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l<
> https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l>
>
> On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>>
> wrote:
>
> > Hi Justin,
> >
> > Forgive me but I don’t remember a journal persistence issue discussion. I
> > did an email search too and did not see that discussion either. Would you
> > please refresh me on the discussion? I may have inadvertently deleted
> that?
> >
> > I do see data being replicated to my production slave/standby instance on
> > initial startup of the two broker instances. Failover works as expected.
> > But when the primary instance starts backup I do not see the active slave
> > replicate to the starting primary and I do not see the active slave
> restart
> > and go to the backup server state.
> >
> > What is the communication sequence when the primary starts up after it
> > quit with the active backup broker? In other words, how is the active
> > backup broker notified to start replication to the newly started primary
> > broker?
> >
> > I ran the replicated-failback example again with TRACE level logging
> > turned on and I do see the attemptFailback=false message in the
> > target/server1/log/artemis.log file at startup, so it must be changed
> later
> > or am I not understanding the purpose of this flag?
> >
> > 2024-01-12 14:40:18,341 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> > discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=localhost, b=null],
> > nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=localhost)
> >
> > From my production slave broker log
> >
> >
> > 024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > Thanks,
> > Paul
> > From: Justin Bertram <jb...@apache.org>>
> > Date: Friday, January 12, 2024 at 3:38 PM
> > To: users@activemq.apache.org<ma...@activemq.apache.org> <
> users@activemq.apache.org<ma...@activemq.apache.org>>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > > Is it possible to use jgroups instead of UDP for the cluster connection
> > in the example?
> >
> > Yes. Check out examples/features/clustered/clustered-jgroups for an
> example
> > configuration. Keep in mind that UDP, JGroups, etc. is only used for
> > initial discovery. Once brokers are discovered then normal TCP
> connections
> > are created.
> >
> > Before I continue looking through your logs, etc. can you confirm that
> you
> > resolved the journal persistence issue I described in my previous email?
> If
> > that's not resolved there's no chance for failback to occur.
> >
> >
> > Justin
> >
> >
> >
> > On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> > paul.shields@hpe.com<ma...@hpe.com>>
> > wrote:
> >
> > > Hi Justin,
> > >
> > > I tried the example and it works. I turned up logging to get a success
> > > path through the code to maybe see where things get de-railed. Is it
> > > possible to use jgroups instead of UDP for the cluster connection in
> the
> > > example? That would give me a better success signature.
> > >
> > > My standby instance has this in the broker.xml which is the same as in
> > the
> > > example:
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > I am seeing this series of log messages that suggests that failback is
> > not
> > > enabled. Is the attemptFailback=false the default at this point in
> > startup
> > > and it is changed later or it there something gone wrong at this point?
> > >
> > > 2024-01-11 18:24:14,066 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.Topology]
> Topology@7e0a913c
> > [owner=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > >
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > > QuorumManager(server=null) about node up =
> > > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > > Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null]
> > >
> > > 2024-01-11 18:24:14,067 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.Topology]
> Topology@7e0a913c
> > [owner=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > >
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > >
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> <mailto:
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> >
> > > <mailto:
> <mailto:%0b>> >
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> <mailto:
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> >
> > >
> > > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > > Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null]
> > >
> > > 2024-01-11 18:24:14,067 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > ClientSessionFactoryImpl received backup update for live/backup pair =
> > > TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > > / null but it didn't belong to TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > >
> > > 2024-01-11 18:24:14,067 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Cluster Connected
> > >
> > > 2024-01-11 18:24:14,068 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Starting backup manager
> > >
> > > 2024-01-11 18:24:14,069 DEBUG
> > > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > > backup config ClusterConnectionConfiguration{name='my-cluster',
> > address='',
> > > connectorName='artemis', clientFailureCheckPeriod=30000,
> > > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > > maxRetryInterval=32000, initialConnectAttempts=20,
> reconnectAttempts=-1,
> > > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > > staticConnectors=[], discoveryGroupName='my-discovery-group',
> maxHops=1,
> > > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > > clusterNotificationAttempts=2}
> > >
> > > 2024-01-11 18:24:14,081 DEBUG
> > > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > > BackupManager connecting to DiscoveryBackupConnector
> > > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> > >
> > > 2024-01-11 18:24:14,082 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Starting Backup Server
> > >
> > > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Setting server state as started
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > looking up the node through nodeLocator.locateNode()
> > >
> > > 2024-01-11 18:24:14,083 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Connecting towards a possible live, connection
> > > information=Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> > >
> > > 2024-01-11 18:24:14,083 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Calling
> > >
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> > >
> > > java.lang.Exception:trace<ftp://java.lang.Exception/trace>
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1
> <
> https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0<
> ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1
> <
> https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0<
> ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1
> <
> https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0<
> ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1
> <
> https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0<
> ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1
> <
> https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0<
> ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1
> <
> https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0<
> ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1
> <
> https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1
> >(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4
> rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<
> https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0<
> ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1
> <
> https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1
> >(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1
> <
> https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0<
> ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > Trying reconnection attempt 0/0
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > Trying to connect with
> > >
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> <mailto:connectorFactory
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18>
> > > <mailto:connectorFactory<mailto:connectorFactory>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> <mailto:
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> >
> > >
> > > and currentConnectorConfig: TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Connector NettyConnector
> > > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> > useServlet=false,
> > > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > > using native epoll
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Started EPOLL Netty Connector version 4.1.86.Final to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<
> ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616
> >
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Remote destination:
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.25:61616<ftp://10.44.0.25/61616>
> > >
> > >
> > > Thanks,
> > > Paul
> > >
> > > From: Justin Bertram <jb...@apache.org>>
> > > Date: Wednesday, December 13, 2023 at 3:23 PM
> > > To: users@activemq.apache.org<ma...@activemq.apache.org> <
> users@activemq.apache.org<ma...@activemq.apache.org>>
> > > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > > failback
> > > ActiveMQ Artemis ships with an example in the
> > > examples/features/ha/replicated-failback directory that demonstrates
> how
> > > this works. I ran this example and it worked fine. I also started up
> the
> > > brokers manually and triggered failover and failback. Again, everything
> > > worked fine. My only guess is that there's something configured wrong
> or
> > > perhaps there's an environmental issue that's preventing failback from
> > > working properly for you. If you could provide clear steps on how to
> > > reproduce this issue I could investigate further.
> > >
> > >
> > > Justin
> > >
> > > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> > paul.shields@hpe.com<ma...@hpe.com>
> > > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I am having an issue with failback to the master broker. I start the
> > > > master broker then the slave broker. I see the replication from the
> > > master
> > > > to the slave happen. I stop the master/active broker and a failover
> > > happens
> > > > to the slave as expected. When the master is started a failback to it
> > is
> > > > expected but does not happen. The following is my ha-policy config.
> > > Did I
> > > > miss something in the ha-policy configuration? We are using version
> > > 2.28.0
> > > > and java17.
> > > >
> > > > Thanks,
> > > > Paul
> > > >
> > > >
> > > > Master broker.xml config
> > > >
> > > > <ha-policy>
> > > >
> > > > <replication>
> > > >
> > > > <master>
> > > >
> > > > <check-for-live-server>true</check-for-live-server>
> > > >
> > > > </master>
> > > >
> > > > </replication>
> > > >
> > > > </ha-policy>
> > > >
> > > > Slave broker.xml config
> > > >
> > > > <ha-policy>
> > > >
> > > > <replication>
> > > >
> > > > <slave>
> > > >
> > > > <allow-failback>true</allow-failback>
> > > >
> > > > </slave>
> > > >
> > > > </replication>
> > > >
> > > > </ha-policy>
> > > >
> > > > The following are the logs from the master and slave brokers
> > > > HA “master” broker log after restart
> > > > 2023-12-08 20:42:57,509 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<
> ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883
> >
> > for
> > > > protocols [MQTT]
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > >
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<
> ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616
> >
> > for
> > > > protocols [CORE]
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221007: Server is now live
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > > Starting embedded web server
> > > > 2023-12-08 20:42:57,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 15, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,627 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3
> > > > 2023-12-08 20:42:57,657 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> <mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2>
> > > > [ID=47edf16f<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> <mailto:
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> >
> > > [ID=47edf16f>,
> > > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/
> 10.44.0.18:34508<http://10.44.0.18:34508>]]
> > > > 2023-12-08 20:42:57,720 INFO
> > > > [org.apache.activemq.hawtio.branding.PluginContextListener]
> Initialized
> > > > activemq-branding plugin
> > > > 2023-12-08 20:42:57,737 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is connected
> > > > 2023-12-08 20:42:57,755 INFO
> > > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > > artemis-plugin plugin
> > > > 2023-12-08 20:42:57,766 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 18, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,767 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 20, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > > Initialising
> > > > hawtio services
> > > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > > Configuration
> > > > will be discovered via system properties
> > > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > > Hawtio 2.15.0
> > > > 2023-12-08 20:42:57,975 INFO
> > > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > > authentication filter, JAAS realm: "activemq" authorized role(s):
> > "admin"
> > > > role principal classes:
> > > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > > servlet is disabled
> > > > 2023-12-08 20:42:57,989 INFO
> > > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia
> overridden
> > > > property: [key=policyLocation,
> > > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > > HTTP Server started at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > >
> > > >
> > > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > > Artemis Jolokia REST API available at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > >
> > > >
> > > > 2023-12-08
> > > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08>
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08>
> > >
> > > >
> > > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > > Console
> > > > available at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > >
> > > >
> > > > 2023-12-08
> > > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> > >
> > > >
> > > > 20:42:58,455 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl]
> > > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> <mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d>
> > > > [ID=f85ec51b<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> <mailto:
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> >
> > > [ID=f85ec51b>,
> > > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/
> 127.0.0.6:49865<http://127.0.0.6:49865>]]
> > > > 2023-12-08 20:42:59,524 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> <mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858>
> > > > [ID=7cef23a2<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> <mailto:
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> >
> > > [ID=7cef23a2>,
> > > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/
> 127.0.0.6:59691<http://127.0.0.6:59691>]]
> > > > 2023-12-08 20:42:59,535 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 22, result: NO_BINDINGS
> > > > 2023-12-08 20:42:59,536 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> > connectionID:
> > > > 7cef23a2
> > > > 2023-12-08 20:42:59,549 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 24, result: NO_BINDINGS
> > > > 2023-12-08 20:42:59,550 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> > connectionID:
> > > > 7cef23a2
> > > > 2023-12-08 20:42:59,583 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 26, result: OK
> > > > 2023-12-08 20:42:59,584 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=true]@5f46053b
> > > > 2023-12-08 20:42:59,593 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> > created
> > > > consumer with ID: 0, with session name:
> > > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > > 2023-12-08 20:42:59,594 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 28, result: OK
> > > > 2023-12-08 20:42:59,616 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > > 2023-12-08 20:42:59,615 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085,
> > > > with transaction: null
> > > > 2023-12-08 20:42:59,621 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 31, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,621 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
> > > > with transaction: null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 32, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
> > > > with transaction: null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 35, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293,
> > > > with transaction: null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 39, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:43:08,447 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> <mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256>
> > > > [ID=1d1d79ce<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> <mailto:
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> >
> > > [ID=1d1d79ce>,
> > > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/
> 127.0.0.6:60787<http://127.0.0.6:60787>]]
> > > > 2023-12-08 20:43:08,464 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> <mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7>
> > > > [ID=8f88c6d5<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> <mailto:
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> >
> > > [ID=8f88c6d5>,
> > > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/
> 127.0.0.6:40065<http://127.0.0.6:40065>]]
> > > >
> > > >
> > > > HA “slave” broker log.
> > > >
> > > > 2023-12-08 20:37:58,374 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0
> [null]
> > > > started, waiting live to fail before it gets active
> > > >
> > > > 2023-12-08 20:37:59,020 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > > synchronized with live server,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > > >
> > > > 2023-12-08 20:38:00,032 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221031: backup announced
> > > > 2023-12-08 20:42:27,415 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221084: Requested 0 quorum votes
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221071: Failing over based on quorum vote results.
> > > > 2023-12-08 20:42:27,431 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212037: Connection failure to
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected:
> AMQ219015: The connection was
> > > > disconnected because of server shutdown [code=DISCONNECTED]
> > > > 2023-12-08 20:42:27,431 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212037: Connection failure to
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected:
> AMQ219015: The connection was
> > > > disconnected because of server shutdown [code=DISCONNECTED]
> > > > 2023-12-08 20:42:27,434 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > > 2023-12-08 20:42:27,574 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > > 2023-12-08 20:42:27,575 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > > 2023-12-08 20:42:27,578 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > > 2023-12-08 20:42:27,578 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > > 2023-12-08 20:42:27,611 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221007: Server is now live
> > > > 2023-12-08 20:42:27,699 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<
> ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883
> >
> > for
> > > > protocols [MQTT]
> > > > 2023-12-08 20:42:27,717 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > >
> cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<
> ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616
> >
> > for
> > > > protocols [CORE]
> > > > 2023-12-08 20:42:51,946 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > [ID=df357d0e>,
> > > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/
> 127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > > 2023-12-08 20:42:56,977 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > > destroyed
> > > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > [ID=df357d0e>,
> > > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/
> 127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > > 2023-12-08 20:42:57,443 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212004: Failed to connect to server.
> > > > 2023-12-08 20:42:57,445 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 26, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,447 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6
> > > > 2023-12-08 20:42:57,512 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > > failure detected. Unblocking a blocking call that will never get a
> > > response
> > > > 2023-12-08 20:42:57,513 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is unable to connect to
> destination.
> > > > Retrying
> > > > 2023-12-08 20:42:57,629 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > > > [ID=483170f6<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > > [ID=483170f6>,
> > > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/
> 127.0.0.6:55911<http://127.0.0.6:55911>]]
> > > > 2023-12-08 20:42:57,630 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > > > [ID=4514a2b3<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > > [ID=4514a2b3>,
> > > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/
> 127.0.0.6:60199<http://127.0.0.6:60199>]]
> > > > 2023-12-08 20:42:57,648 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > > > [ID=fcde4d78<mailto:transportConnection<mailto:transportConnection>
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > > [ID=fcde4d78>,
> > > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/
> 127.0.0.6:53221<http://127.0.0.6:53221>]]
> > > > 2023-12-08 20:42:57,663 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 29, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,663 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> > connectionID:
> > > > fcde4d78
> > > > 2023-12-08 20:42:57,679 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 31, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,679 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> > connectionID:
> > > > fcde4d78
> > > > 2023-12-08 20:42:57,708 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 33, result: OK
> > > > 2023-12-08 20:42:57,709 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=true]@3645037f
> > > > 2023-12-08 20:42:57,719 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> > created
> > > > consumer with ID: 0, with session name:
> > > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > > 2023-12-08 20:42:57,720 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 35, result: OK
> > > > 2023-12-08 20:42:57,755 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > > 2023-12-08 20:42:57,753 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563,
> > > > with transaction: null
> > > > 2023-12-08 20:42:57,760 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 38, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,761 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
> > > > with transaction: null
> > > > 2023-12-08 20:42:57,761 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 41, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
> > > > with transaction: null
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 43, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555,
> > > > with transaction: null
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 44, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:59,607 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is connected
> > > > 2023-12-08 20:42:59,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 46, result: OK
> > > > 2023-12-08 20:42:59,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 48, result: OK
> > > > 2023-12-08 20:43:07,437 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > > failure detected. Unblocking a blocking call that will never get a
> > > response
> > > >
> > > >
> > > >
> > >
> >
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by "Shields, Paul Michael" <pa...@hpe.com>.
Hi Justin,
A follow-on question about persistent data and failover. The documentation makes this statement. “When using replication, the live and the backup servers do not share the same data directories, all data synchronization is done over the network. Therefore all (persistent) data received by the live server will be duplicated to the backup.” [1]. We are using the MQTT protocol. We have published messages that have the retained flag set. We also use “last will” messages and set the lastWillRetain flag to true. When the last will message is sent by the broker the retained message for the publisher is updated with the last will message. Now for the question. Are last will messages considered persistent data and are duplicated to the backup or is the retained message updated with the last will message duplicated to the backup broker? We seem to be losing last will messages on a broker failover (ie the retained message does not contain the last will message).
Thanks,
Paul
[1]: https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies
From: Justin Bertram <jb...@apache.org>
Date: Friday, January 19, 2024 at 11:47 AM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
> So even though the persistent data will be stored pod-local...
If you disable persistence on the broker then no data will be persisted on the local pod.
> ...enabling persistence has the knock-on effect of also making it available to the backup server in addition to whatever the “replication” model does?
Only persistent data is replicated to the backup. This is noted in the documentation [1].
To be clear, this is not "in addition" to what replication is doing. The replication manager "listens" for persisted data and that's the information that it sends to the backup. If you disable persistence then no data is replicated.
Generally speaking, client applications make a basic decision about the messages they send - persistent or non-persistent. Sending persistent messages comes with certain reliability guarantees but worse performance. Sending non-persistent messages comes with better performance but less reliability. The ability to survive a failure is one of the reliability benefits enjoyed by persistent messages.
Justin
[1] https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies>
On Fri, Jan 19, 2024 at 10:50 AM John Lilley <jo...@redpointglobal.com.invalid> wrote:
Hi Justin,
This does seem counter-intuitive. We’re using replication, not shared storage. So even though the persistent data will be stored pod-local, enabling persistence has the knock-on effect of also making it available to the backup server in addition to whatever the “replication” model does?
john
[rg]<https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Friday, January 19, 2024 9:42 AM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> ...do we currently set persistenceEnable=false? I would expect there’s no reason for setting it true, since the pod-local storage will evaporate with the pod.
You definitely want to enable persistence because only persistent data will be available on the backup.
Justin
On Fri, Jan 19, 2024 at 10:21 AM John Lilley <jo...@redpointglobal.com.invalid>> wrote:
Hi Paul,
Thanks for the information! It sounds like you have experience with AMQ in K8S. Can you share your two broker.xml files with us?
Lino: do we currently set persistenceEnable=false? I would expect there’s no reason for setting it true, since the pod-local storage will evaporate with the pod.
Thanks
John
Error! Filename not specified.<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Shields, Paul Michael <pa...@hpe.com>>
Sent: Friday, January 19, 2024 8:51 AM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hi John,
The <node-manager-lock-directory>. Needs to be specified in both the master and slave broker.xml files.
the nodeUUID which is contained in the server lock file was changing between active broker startups. The server.lock file is stored in the /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our current configuration of the broker has persistenceEnable=false which causes any data/broker state to be removed when the broker pod is destroyed. Thus a new nodeUUID was being generated on each Primary broker startup. When the Primary broker instance starts up it looks for an existing server.lock file. When it does not find one it generates a new nodeUUID and places it in the server.lock file, otherwise it uses the nodeUUID contained within the existing server.lock file. The Primary broker then communicates this nodeUUID through the cluster connection to the Standby broker instance on startup. The Standby broker instance uses the nodeUUID to determine if the Primary broker instance is part of its HA pair. The Standby broker instance saved off the original nodeUUID when the HA cluster was first started and since the recently communicated nodeUUID does not match the nodeUUID that it previously stored the fail back is not performed. Once persistence was enabled the Standby broker instance performed the fallback to the Primary instance and was no longer in the "active" state
This is what I have observed in our Kubernetes environment. Justin please make any corrections to my description of the nodeUUID use.
Paul
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Date: Thursday, January 18, 2024 at 5:49 PM
To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Justin,
Can you elaborate on the lock file in the <node-manager-lock-directory>?
Does it need to be specified in the broker.xml for both the master and the slave?
Does it actually get used as a “lock” in the concurrency sense, to coordinate the live/backup election?
Thanks
john
Error! Filename not specified.<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,0rYuECbqGwpo9IjFKJrQQrfyzL5V_aEiqSXV25I7DeF4wuQn3I6z1sJCY1NkNJ-bjdpV8nVix_VDU0mK5tgrxuSQaSyrlVLxE5EiTf0Z8G4F&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Monday, January 15, 2024 9:56 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
In the replication case you still need a PV in K8s, but I don't believe you need to put the whole journal on the PV (and suffer the performance penalty). You should just need to point the node-manager-lock-directory in broker.xml to the PV.
Justin
On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
Based on your reply in the provided link, are you saying that even for the replication HA case, you still need an external FileShare PV, in K8s, so that the nodeID can be persisted between restarts. In this use case, is the performance of this FileShare less of a concern than it is for the shared-store use case?
Thanks,
Lino
Error! Filename not specified.<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
-----Original Message-----
From: Justin Bertram <jb...@apache.org>>
Sent: Friday, January 12, 2024 4:53 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
I sent that email on December 15. You can find it on the web-based mailing
list interface [1].
Justin
[1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l<https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l>
On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <pa...@hpe.com>>
wrote:
> Hi Justin,
>
> Forgive me but I don’t remember a journal persistence issue discussion. I
> did an email search too and did not see that discussion either. Would you
> please refresh me on the discussion? I may have inadvertently deleted that?
>
> I do see data being replicated to my production slave/standby instance on
> initial startup of the two broker instances. Failover works as expected.
> But when the primary instance starts backup I do not see the active slave
> replicate to the starting primary and I do not see the active slave restart
> and go to the backup server state.
>
> What is the communication sequence when the primary starts up after it
> quit with the active backup broker? In other words, how is the active
> backup broker notified to start replication to the newly started primary
> broker?
>
> I ran the replicated-failback example again with TRACE level logging
> turned on and I do see the attemptFailback=false message in the
> target/server1/log/artemis.log file at startup, so it must be changed later
> or am I not understanding the purpose of this flag?
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> discoveryInitialWaitTimeout=10000}]
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost, b=null],
> nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost)
>
> From my production slave broker log
>
>
> 024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> Thanks,
> Paul
> From: Justin Bertram <jb...@apache.org>>
> Date: Friday, January 12, 2024 at 3:38 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > Is it possible to use jgroups instead of UDP for the cluster connection
> in the example?
>
> Yes. Check out examples/features/clustered/clustered-jgroups for an example
> configuration. Keep in mind that UDP, JGroups, etc. is only used for
> initial discovery. Once brokers are discovered then normal TCP connections
> are created.
>
> Before I continue looking through your logs, etc. can you confirm that you
> resolved the journal persistence issue I described in my previous email? If
> that's not resolved there's no chance for failback to occur.
>
>
> Justin
>
>
>
> On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>>
> wrote:
>
> > Hi Justin,
> >
> > I tried the example and it works. I turned up logging to get a success
> > path through the code to maybe see where things get de-railed. Is it
> > possible to use jgroups instead of UDP for the cluster connection in the
> > example? That would give me a better success signature.
> >
> > My standby instance has this in the broker.xml which is the same as in
> the
> > example:
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > I am seeing this series of log messages that suggests that failback is
> not
> > enabled. Is the attemptFailback=false the default at this point in
> startup
> > and it is changed later or it there something gone wrong at this point?
> >
> > 2024-01-11 18:24:14,066 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > QuorumManager(server=null) about node up =
> > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> > <mailto:
<mailto:%0b>> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> >
> > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > ClientSessionFactoryImpl received backup update for live/backup pair =
> > TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > / null but it didn't belong to TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,067 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Cluster Connected
> >
> > 2024-01-11 18:24:14,068 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting backup manager
> >
> > 2024-01-11 18:24:14,069 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > backup config ClusterConnectionConfiguration{name='my-cluster',
> address='',
> > connectorName='artemis', clientFailureCheckPeriod=30000,
> > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > clusterNotificationAttempts=2}
> >
> > 2024-01-11 18:24:14,081 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > 2024-01-11 18:24:14,083 TRACE
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> >
> > java.lang.Exception:trace<ftp://java.lang.Exception/trace>
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4 rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying reconnection attempt 0/0
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18> > > <mailto:connectorFactory<mailto:connectorFactory>
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18>
> >
> > and currentConnectorConfig: TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Connector NettyConnector
> > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using native epoll
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Started EPOLL Netty Connector version 4.1.86.Final to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Remote destination:
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.25:61616<ftp://10.44.0.25/61616>
> >
> >
> > Thanks,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>>
> > Date: Wednesday, December 13, 2023 at 3:23 PM
> > To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > ActiveMQ Artemis ships with an example in the
> > examples/features/ha/replicated-failback directory that demonstrates how
> > this works. I ran this example and it worked fine. I also started up the
> > brokers manually and triggered failover and failback. Again, everything
> > worked fine. My only guess is that there's something configured wrong or
> > perhaps there's an environmental issue that's preventing failback from
> > working properly for you. If you could provide clear steps on how to
> > reproduce this issue I could investigate further.
> >
> >
> > Justin
> >
> > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am having an issue with failback to the master broker. I start the
> > > master broker then the slave broker. I see the replication from the
> > master
> > > to the slave happen. I stop the master/active broker and a failover
> > happens
> > > to the slave as expected. When the master is started a failback to it
> is
> > > expected but does not happen. The following is my ha-policy config.
> > Did I
> > > miss something in the ha-policy configuration? We are using version
> > 2.28.0
> > > and java17.
> > >
> > > Thanks,
> > > Paul
> > >
> > >
> > > Master broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <master>
> > >
> > > <check-for-live-server>true</check-for-live-server>
> > >
> > > </master>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > Slave broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > The following are the logs from the master and slave brokers
> > > HA “master” broker log after restart
> > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > Starting embedded web server
> > > 2023-12-08 20:42:57,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 15, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,627 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3
> > > 2023-12-08 20:42:57,657 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2> > > > [ID=47edf16f<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2>
> > [ID=47edf16f>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/10.44.0.18:34508<http://10.44.0.18:34508>]]
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > > activemq-branding plugin
> > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > artemis-plugin plugin
> > > 2023-12-08 20:42:57,766 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 18, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,767 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 20, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > Initialising
> > > hawtio services
> > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > Configuration
> > > will be discovered via system properties
> > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > Hawtio 2.15.0
> > > 2023-12-08 20:42:57,975 INFO
> > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > authentication filter, JAAS realm: "activemq" authorized role(s):
> "admin"
> > > role principal classes:
> > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > servlet is disabled
> > > 2023-12-08 20:42:57,989 INFO
> > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > > property: [key=policyLocation,
> > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > HTTP Server started at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> >
> > >
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > Artemis Jolokia REST API available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08> > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08> > >
> > >
> > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > Console
> > > available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08>
> >
> > >
> > > 20:42:58,455 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl]
> > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d> > > > [ID=f85ec51b<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d>
> > [ID=f85ec51b>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:49865<http://127.0.0.6:49865>]]
> > > 2023-12-08 20:42:59,524 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858> > > > [ID=7cef23a2<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858>
> > [ID=7cef23a2>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:59691<http://127.0.0.6:59691>]]
> > > 2023-12-08 20:42:59,535 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 22, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,536 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,549 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 24, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,550 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,583 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: OK
> > > 2023-12-08 20:42:59,584 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@5f46053b
> > > 2023-12-08 20:42:59,593 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > 2023-12-08 20:42:59,594 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 28, result: OK
> > > 2023-12-08 20:42:59,616 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:59,615 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 31, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 32, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 35, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 39, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:43:08,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256> > > > [ID=1d1d79ce<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256>
> > [ID=1d1d79ce>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:60787<http://127.0.0.6:60787>]]
> > > 2023-12-08 20:43:08,464 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7> > > > [ID=8f88c6d5<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7>
> > [ID=8f88c6d5>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:40065<http://127.0.0.6:40065>]]
> > >
> > >
> > > HA “slave” broker log.
> > >
> > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > synchronized with live server,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > >
> > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221031: backup announced
> > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221084: Requested 0 quorum votes
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221071: Failing over based on quorum vote results.
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:51,946 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:56,977 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > destroyed
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2023-12-08 20:42:57,445 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6
> > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > > Retrying
> > > 2023-12-08 20:42:57,629 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > > [ID=483170f6<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:55911<http://127.0.0.6:55911>]]
> > > 2023-12-08 20:42:57,630 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > > [ID=4514a2b3<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:60199<http://127.0.0.6:60199>]]
> > > 2023-12-08 20:42:57,648 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > > [ID=fcde4d78<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:53221<http://127.0.0.6:53221>]]
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 29, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 31, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,708 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 33, result: OK
> > > 2023-12-08 20:42:57,709 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@3645037f
> > > 2023-12-08 20:42:57,719 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 35, result: OK
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:57,753 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > 2023-12-08 20:42:57,760 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 38, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, > > > with transaction: null
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 41, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 43, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 44, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 46, result: OK
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 48, result: OK
> > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > >
> > >
> > >
> >
>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
RE: Trouble with Replication HA Master/Slave config performing failback
Posted by John Lilley <jo...@redpointglobal.com.INVALID>.
Thanks for the explanation!
john
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>
Sent: Friday, January 19, 2024 10:47 AM
To: users@activemq.apache.org
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> So even though the persistent data will be stored pod-local...
If you disable persistence on the broker then no data will be persisted on the local pod.
> ...enabling persistence has the knock-on effect of also making it available to the backup server in addition to whatever the “replication” model does?
Only persistent data is replicated to the backup. This is noted in the documentation [1].
To be clear, this is not "in addition" to what replication is doing. The replication manager "listens" for persisted data and that's the information that it sends to the backup. If you disable persistence then no data is replicated.
Generally speaking, client applications make a basic decision about the messages they send - persistent or non-persistent. Sending persistent messages comes with certain reliability guarantees but worse performance. Sending non-persistent messages comes with better performance but less reliability. The ability to survive a failure is one of the reliability benefits enjoyed by persistent messages.
Justin
[1] https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies
On Fri, Jan 19, 2024 at 10:50 AM John Lilley <jo...@redpointglobal.com.invalid>> wrote:
Hi Justin,
This does seem counter-intuitive. We’re using replication, not shared storage. So even though the persistent data will be stored pod-local, enabling persistence has the knock-on effect of also making it available to the backup server in addition to whatever the “replication” model does?
john
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,lxHhM5LAitcPy-8xwp0ssDjxpiHpfZ1Vj_Y_WbfqmMyaVViZov1xp1JdKdzkuZ1lM_3M6NrhK93HXnDVh7MnA2DEsO-DZbrOuou-nq_cRDDW7rQ0bhcr4g,,&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Friday, January 19, 2024 9:42 AM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> ...do we currently set persistenceEnable=false? I would expect there’s no reason for setting it true, since the pod-local storage will evaporate with the pod.
You definitely want to enable persistence because only persistent data will be available on the backup.
Justin
On Fri, Jan 19, 2024 at 10:21 AM John Lilley <jo...@redpointglobal.com.invalid>> wrote:
Hi Paul,
Thanks for the information! It sounds like you have experience with AMQ in K8S. Can you share your two broker.xml files with us?
Lino: do we currently set persistenceEnable=false? I would expect there’s no reason for setting it true, since the pod-local storage will evaporate with the pod.
Thanks
John
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
John Lilley <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
Data Management Chief Architect, Redpoint Global Inc. <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
34 Washington Street, Suite 205 Wellesley Hills, MA 02481 <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
M: +1 7209385761 | john.lilley@redpointglobal.com <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
From: Shields, Paul Michael <pa...@hpe.com>
Sent: Friday, January 19, 2024 8:51 AM
To: users@activemq.apache.org
Subject: Re: Trouble with Replication HA Master/Slave config performing failback<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
Hi John,<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
The <node-manager-lock-directory>. Needs to be specified in both the master and slave broker.xml files.<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
the nodeUUID which is contained in the server lock file was changing between active broker startups. The server.lock file is stored in the /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our current configuration of the broker has persistenceEnable=false which causes any data/broker state to be removed when the broker pod is destroyed. Thus a new nodeUUID was being generated on each Primary broker startup. When the Primary broker instance starts up it looks for an existing server.lock file. When it does not find one it generates a new nodeUUID and places it in the server.lock file, otherwise it uses the nodeUUID contained within the existing server.lock file. The Primary broker then communicates this nodeUUID through the cluster connection to the Standby broker instance on startup. The Standby broker instance uses the nodeUUID to determine if the Primary broker instance is part of its HA pair. The Standby broker instance saved off the original nodeUUID when the HA cluster was first started and since the recently communicated nodeUUID does not match the nodeUUID that it previously stored the fail back is not performed. Once persistence was enabled the Standby broker instance performed the fallback to the Primary instance and was no longer in the "active" state <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
This is what I have observed in our Kubernetes environment. Justin please make any corrections to my description of the nodeUUID use.<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
Paul<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
From: John Lilley <jo...@redpointglobal.com.INVALID>
Date: Thursday, January 18, 2024 at 5:49 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
Hi Justin,<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
Can you elaborate on the lock file in the <node-manager-lock-directory>?<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
Does it need to be specified in the broker.xml for both the master and the slave?<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
Does it actually get used as a “lock” in the concurrency sense, to coordinate the live/backup election?<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
Thanks<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
john<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
John Lilley <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
Data Management Chief Architect, Redpoint Global Inc. <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
34 Washington Street, Suite 205 Wellesley Hills, MA 02481 <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
M: +1 7209385761 | john.lilley@redpointglobal.com <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
From: Justin Bertram <jb...@apache.org>
Sent: Monday, January 15, 2024 9:56 PM
To: users@activemq.apache.org
Subject: Re: Trouble with Replication HA Master/Slave config performing failback<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
In the replication case you still need a PV in K8s, but I don't believe you need to put the whole journal on the PV (and suffer the performance penalty). You should just need to point the node-manager-lock-directory in broker.xml to the PV.<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
Justin<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <li...@redpointglobal.com.invalid> wrote:<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
Hi Justin, <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
Based on your reply in the provided link, are you saying that even for the replication HA case, you still need an external FileShare PV, in K8s, so that the nodeID can be persisted between restarts. In this use case, is the performance of this FileShare less of a concern than it is for the shared-store use case? <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
Thanks,
Lino <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
Lino Pereira <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
C++ Developer, Redpoint Global Inc. <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
34 Washington Street, Suite 205 Wellesley Hills, MA 02481 <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
lino.pereira@redpointglobal.com <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
-----Original Message-----
From: Justin Bertram <jb...@apache.org>
Sent: Friday, January 12, 2024 4:53 PM
To: users@activemq.apache.org
Subject: Re: Trouble with Replication HA Master/Slave config performing failback <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. *** <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
I sent that email on December 15. You can find it on the web-based mailing
list interface [1]. <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
Justin <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
[1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <pa...@hpe.com>
wrote: <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
> Hi Justin,
>
> Forgive me but I don’t remember a journal persistence issue discussion. I
> did an email search too and did not see that discussion either. Would you
> please refresh me on the discussion? I may have inadvertently deleted that?
>
> I do see data being replicated to my production slave/standby instance on
> initial startup of the two broker instances. Failover works as expected.
> But when the primary instance starts backup I do not see the active slave
> replicate to the starting primary and I do not see the active slave restart
> and go to the backup server state.
>
> What is the communication sequence when the primary starts up after it
> quit with the active backup broker? In other words, how is the active
> backup broker notified to start replication to the newly started primary
> broker?
>
> I ran the replicated-failback example again with TRACE level logging
> turned on and I do see the attemptFailback=false message in the
> target/server1/log/artemis.log file at startup, so it must be changed later
> or am I not understanding the purpose of this flag?
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> discoveryInitialWaitTimeout=10000}]
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost, b=null],
> nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost)
>
> From my production slave broker log
>
>
> 024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> Thanks,
> Paul
> From: Justin Bertram <jb...@apache.org>
> Date: Friday, January 12, 2024 at 3:38 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > Is it possible to use jgroups instead of UDP for the cluster connection
> in the example?
>
> Yes. Check out examples/features/clustered/clustered-jgroups for an example
> configuration. Keep in mind that UDP, JGroups, etc. is only used for
> initial discovery. Once brokers are discovered then normal TCP connections
> are created.
>
> Before I continue looking through your logs, etc. can you confirm that you
> resolved the journal persistence issue I described in my previous email? If
> that's not resolved there's no chance for failback to occur.
>
>
> Justin
>
>
>
> On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> paul.shields@hpe.com>
> wrote:
>
> > Hi Justin,
> >
> > I tried the example and it works. I turned up logging to get a success
> > path through the code to maybe see where things get de-railed. Is it
> > possible to use jgroups instead of UDP for the cluster connection in the
> > example? That would give me a better success signature.
> >
> > My standby instance has this in the broker.xml which is the same as in
> the
> > example:
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > I am seeing this series of log messages that suggests that failback is
> not
> > enabled. Is the attemptFailback=false the default at this point in
> startup
> > and it is changed later or it there something gone wrong at this point?
> >
> > 2024-01-11 18:24:14,066 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > QuorumManager(server=null) about node up =
> > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> > <mailto:
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> >
> > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > ClientSessionFactoryImpl received backup update for live/backup pair =
> > TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > / null but it didn't belong to TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,067 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Cluster Connected
> >
> > 2024-01-11 18:24:14,068 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting backup manager
> >
> > 2024-01-11 18:24:14,069 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > backup config ClusterConnectionConfiguration{name='my-cluster',
> address='',
> > connectorName='artemis', clientFailureCheckPeriod=30000,
> > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > clusterNotificationAttempts=2}
> >
> > 2024-01-11 18:24:14,081 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > 2024-01-11 18:24:14,083 TRACE
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> >
> > java.lang.Exception:trace
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1) > > ~[artemis-core-client-2.28.0.jar:2.28.0]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1) > > ~[artemis-core-client-2.28.0.jar:2.28.0]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1) > > ~[artemis-core-client-2.28.0.jar:2.28.0]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1) > > ~[artemis-core-client-2.28.0.jar:2.28.0]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1) > > ~[artemis-server-2.28.0.jar:2.28.0]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1) > > ~[artemis-server-2.28.0.jar:2.28.0]
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4 rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1) > > ~[artemis-server-2.28.0.jar:2.28.0]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1) > > ~[artemis-server-2.28.0.jar:2.28.0]
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying reconnection attempt 0/0
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18 > > <mailto:connectorFactory
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> >
> > and currentConnectorConfig: TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Connector NettyConnector
> > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using native epoll
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Started EPOLL Netty Connector version 4.1.86.Final to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Remote destination:
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.25:61616
> >
> >
> > Thanks,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>
> > Date: Wednesday, December 13, 2023 at 3:23 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > ActiveMQ Artemis ships with an example in the
> > examples/features/ha/replicated-failback directory that demonstrates how
> > this works. I ran this example and it worked fine. I also started up the
> > brokers manually and triggered failover and failback. Again, everything
> > worked fine. My only guess is that there's something configured wrong or
> > perhaps there's an environmental issue that's preventing failback from
> > working properly for you. If you could provide clear steps on how to
> > reproduce this issue I could investigate further.
> >
> >
> > Justin
> >
> > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> paul.shields@hpe.com
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am having an issue with failback to the master broker. I start the
> > > master broker then the slave broker. I see the replication from the
> > master
> > > to the slave happen. I stop the master/active broker and a failover
> > happens
> > > to the slave as expected. When the master is started a failback to it
> is
> > > expected but does not happen. The following is my ha-policy config.
> > Did I
> > > miss something in the ha-policy configuration? We are using version
> > 2.28.0
> > > and java17.
> > >
> > > Thanks,
> > > Paul
> > >
> > >
> > > Master broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <master>
> > >
> > > <check-for-live-server>true</check-for-live-server>
> > >
> > > </master>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > Slave broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > The following are the logs from the master and slave brokers
> > > HA “master” broker log after restart
> > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > Starting embedded web server
> > > 2023-12-08 20:42:57,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 15, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,627 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3
> > > 2023-12-08 20:42:57,657 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2 > > > [ID=47edf16f<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > [ID=47edf16f>,
> > > local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > > activemq-branding plugin
> > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > artemis-plugin plugin
> > > 2023-12-08 20:42:57,766 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 18, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,767 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 20, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > Initialising
> > > hawtio services
> > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > Configuration
> > > will be discovered via system properties
> > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > Hawtio 2.15.0
> > > 2023-12-08 20:42:57,975 INFO
> > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > authentication filter, JAAS realm: "activemq" authorized role(s):
> "admin"
> > > role principal classes:
> > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > servlet is disabled
> > > 2023-12-08 20:42:57,989 INFO
> > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > > property: [key=policyLocation,
> > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > HTTP Server started at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > >
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > Artemis Jolokia REST API available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > >
> > >
> > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > Console
> > > available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> > >
> > > 20:42:58,455 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl]
> > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d > > > [ID=f85ec51b<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > [ID=f85ec51b>,
> > > local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
> > > 2023-12-08 20:42:59,524 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858 > > > [ID=7cef23a2<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > [ID=7cef23a2>,
> > > local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
> > > 2023-12-08 20:42:59,535 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 22, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,536 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,549 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 24, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,550 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,583 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: OK
> > > 2023-12-08 20:42:59,584 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@5f46053b
> > > 2023-12-08 20:42:59,593 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > 2023-12-08 20:42:59,594 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 28, result: OK
> > > 2023-12-08 20:42:59,616 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:59,615 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 31, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 32, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 35, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 39, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:43:08,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256 > > > [ID=1d1d79ce<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > [ID=1d1d79ce>,
> > > local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
> > > 2023-12-08 20:43:08,464 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7 > > > [ID=8f88c6d5<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > [ID=8f88c6d5>,
> > > local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
> > >
> > >
> > > HA “slave” broker log.
> > >
> > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > synchronized with live server,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > >
> > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221031: backup announced
> > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221084: Requested 0 quorum votes
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221071: Failing over based on quorum vote results.
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:51,946 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > > 2023-12-08 20:42:56,977 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > destroyed
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2023-12-08 20:42:57,445 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6
> > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > > Retrying
> > > 2023-12-08 20:42:57,629 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > > [ID=483170f6<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6>,
> > > local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
> > > 2023-12-08 20:42:57,630 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > > [ID=4514a2b3<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3>,
> > > local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
> > > 2023-12-08 20:42:57,648 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > > [ID=fcde4d78<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78>,
> > > local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 29, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 31, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,708 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 33, result: OK
> > > 2023-12-08 20:42:57,709 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@3645037f
> > > 2023-12-08 20:42:57,719 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 35, result: OK
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:57,753 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > 2023-12-08 20:42:57,760 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 38, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, > > > with transaction: null
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 41, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 43, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 44, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 46, result: OK
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 48, result: OK
> > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > >
> > >
> > >
> >
><https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint. <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint. <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint. <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint. <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by Justin Bertram <jb...@apache.org>.
> So even though the persistent data will be stored pod-local...
If you disable persistence on the broker then no data will be persisted on
the local pod.
> ...enabling persistence has the knock-on effect of also making it
available to the backup server in addition to whatever the “replication”
model does?
Only persistent data is replicated to the backup. This is noted in the
documentation [1].
To be clear, this is not "in addition" to what replication is doing. The
replication manager "listens" for persisted data and that's the information
that it sends to the backup. If you disable persistence then no data is
replicated.
Generally speaking, client applications make a basic decision about the
messages they send - persistent or non-persistent. Sending persistent
messages comes with certain reliability guarantees but worse performance.
Sending non-persistent messages comes with better performance but less
reliability. The ability to survive a failure is one of the reliability
benefits enjoyed by persistent messages.
Justin
[1]
https://activemq.apache.org/components/artemis/documentation/latest/ha.html#ha-policies
On Fri, Jan 19, 2024 at 10:50 AM John Lilley
<jo...@redpointglobal.com.invalid> wrote:
> Hi Justin,
>
>
>
> This does seem counter-intuitive. We’re using replication, not shared
> storage. So even though the persistent data will be stored pod-local,
> enabling persistence has the knock-on effect of also making it available to
> the backup server in addition to whatever the “replication” model does?
>
>
>
> john
>
>
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Friday, January 19, 2024 9:42 AM
> *To:* users@activemq.apache.org
> *Subject:* Re: Trouble with Replication HA Master/Slave config performing
> failback
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > ...do we currently set persistenceEnable=false? I would expect there’s
> no reason for setting it true, since the pod-local storage will evaporate
> with the pod.
>
>
>
> You definitely want to enable persistence because only persistent data
> will be available on the backup.
>
>
>
>
>
> Justin
>
>
>
> On Fri, Jan 19, 2024 at 10:21 AM John Lilley <
> john.lilley@redpointglobal.com.invalid> wrote:
>
> Hi Paul,
>
>
>
> Thanks for the information! It sounds like you have experience with AMQ
> in K8S. Can you share your two broker.xml files with us?
>
>
>
> Lino: do we currently set persistenceEnable=false? I would expect
> there’s no reason for setting it true, since the pod-local storage will
> evaporate with the pod.
>
>
>
> Thanks
>
> John
>
>
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
>
> *John Lilley *
>
> *Data Management Chief Architect, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> *From:* Shields, Paul Michael <pa...@hpe.com>
> *Sent:* Friday, January 19, 2024 8:51 AM
> *To:* users@activemq.apache.org
> *Subject:* Re: Trouble with Replication HA Master/Slave config performing
> failback
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Hi John,
>
>
>
> The <node-manager-lock-directory>. Needs to be specified in both the
> master and slave broker.xml files.
>
>
>
> the nodeUUID which is contained in the server lock file was changing
> between active broker startups. The server.lock file is stored in the
> /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our
> current configuration of the broker has persistenceEnable=false which
> causes any data/broker state to be removed when the broker pod is
> destroyed. Thus a new nodeUUID was being generated on each Primary broker
> startup. When the Primary broker instance starts up it looks for an
> existing server.lock file. When it does not find one it generates a new
> nodeUUID and places it in the server.lock file, otherwise it uses the
> nodeUUID contained within the existing server.lock file. The Primary broker
> then communicates this nodeUUID through the cluster connection to the
> Standby broker instance on startup. The Standby broker instance uses the
> nodeUUID to determine if the Primary broker instance is part of its HA
> pair. The Standby broker instance saved off the original nodeUUID when the
> HA cluster was first started and since the recently communicated nodeUUID
> does not match the nodeUUID that it previously stored the fail back is not
> performed. Once persistence was enabled the Standby broker instance
> performed the fallback to the Primary instance and was no longer in the
> "active" state
>
>
>
> This is what I have observed in our Kubernetes environment. Justin please
> make any corrections to my description of the nodeUUID use.
>
>
>
> Paul
>
> *From: *John Lilley <jo...@redpointglobal.com.INVALID>
> *Date: *Thursday, January 18, 2024 at 5:49 PM
> *To: *users@activemq.apache.org <us...@activemq.apache.org>
> *Subject: *RE: Trouble with Replication HA Master/Slave config performing
> failback
>
> Hi Justin,
>
>
>
> Can you elaborate on the lock file in the <node-manager-lock-directory>?
>
> Does it need to be specified in the broker.xml for both the master and the
> slave?
>
> Does it actually get used as a “lock” in the concurrency sense, to
> coordinate the live/backup election?
>
>
>
> Thanks
>
> john
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,0rYuECbqGwpo9IjFKJrQQrfyzL5V_aEiqSXV25I7DeF4wuQn3I6z1sJCY1NkNJ-bjdpV8nVix_VDU0mK5tgrxuSQaSyrlVLxE5EiTf0Z8G4F&typo=1>
>
> *John Lilley *
>
> *Data Management Chief Architect, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Monday, January 15, 2024 9:56 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Trouble with Replication HA Master/Slave config performing
> failback
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> In the replication case you still need a PV in K8s, but I don't believe
> you need to put the whole journal on the PV (and suffer the performance
> penalty). You should just need to point the node-manager-lock-directory in
> broker.xml to the PV.
>
>
>
>
>
> Justin
>
>
>
> On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hi Justin,
>
> Based on your reply in the provided link, are you saying that even for the
> replication HA case, you still need an external FileShare PV, in K8s, so
> that the nodeID can be persisted between restarts. In this use case, is the
> performance of this FileShare less of a concern than it is for the
> shared-store use case?
>
> Thanks,
> Lino
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Friday, January 12, 2024 4:53 PM
> To: users@activemq.apache.org
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
>
> *** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ***
>
> I sent that email on December 15. You can find it on the web-based mailing
> list interface [1].
>
> Justin
>
> [1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l
>
> On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <
> paul.shields@hpe.com>
> wrote:
>
> > Hi Justin,
> >
> > Forgive me but I don’t remember a journal persistence issue discussion. I
> > did an email search too and did not see that discussion either. Would you
> > please refresh me on the discussion? I may have inadvertently deleted
> that?
> >
> > I do see data being replicated to my production slave/standby instance on
> > initial startup of the two broker instances. Failover works as expected.
> > But when the primary instance starts backup I do not see the active slave
> > replicate to the starting primary and I do not see the active slave
> restart
> > and go to the backup server state.
> >
> > What is the communication sequence when the primary starts up after it
> > quit with the active backup broker? In other words, how is the active
> > backup broker notified to start replication to the newly started primary
> > broker?
> >
> > I ran the replicated-failback example again with TRACE level logging
> > turned on and I do see the attemptFailback=false message in the
> > target/server1/log/artemis.log file at startup, so it must be changed
> later
> > or am I not understanding the purpose of this flag?
> >
> > 2024-01-12 14:40:18,341 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> > discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=localhost, b=null],
> > nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=localhost)
> >
> > From my production slave broker log
> >
> >
> > 024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > Thanks,
> > Paul
> > From: Justin Bertram <jb...@apache.org>
> > Date: Friday, January 12, 2024 at 3:38 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > > Is it possible to use jgroups instead of UDP for the cluster connection
> > in the example?
> >
> > Yes. Check out examples/features/clustered/clustered-jgroups for an
> example
> > configuration. Keep in mind that UDP, JGroups, etc. is *only* used for
> > initial discovery. Once brokers are discovered then normal TCP
> connections
> > are created.
> >
> > Before I continue looking through your logs, etc. can you confirm that
> you
> > resolved the journal persistence issue I described in my previous email?
> If
> > that's not resolved there's no chance for failback to occur.
> >
> >
> > Justin
> >
> >
> >
> > On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> > paul.shields@hpe.com>
> > wrote:
> >
> > > Hi Justin,
> > >
> > > I tried the example and it works. I turned up logging to get a success
> > > path through the code to maybe see where things get de-railed. Is it
> > > possible to use jgroups instead of UDP for the cluster connection in
> the
> > > example? That would give me a better success signature.
> > >
> > > My standby instance has this in the broker.xml which is the same as in
> > the
> > > example:
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > I am seeing this series of log messages that suggests that failback is
> > not
> > > enabled. Is the attemptFailback=false the default at this point in
> > startup
> > > and it is changed later or it there something gone wrong at this point?
> > >
> > > 2024-01-11 18:24:14,066 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.Topology]
> Topology@7e0a913c
> > [owner=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > >
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > > QuorumManager(server=null) about node up =
> > > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > > Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null]
> > >
> > > 2024-01-11 18:24:14,067 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.Topology]
> Topology@7e0a913c
> > [owner=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > >
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > >
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> > > <mailto:
> <%0b>> >
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> > >
> > > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > > Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null]
> > >
> > > 2024-01-11 18:24:14,067 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > ClientSessionFactoryImpl received backup update for live/backup pair =
> > > TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > > / null but it didn't belong to TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > >
> > > 2024-01-11 18:24:14,067 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Cluster Connected
> > >
> > > 2024-01-11 18:24:14,068 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Starting backup manager
> > >
> > > 2024-01-11 18:24:14,069 DEBUG
> > > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > > backup config ClusterConnectionConfiguration{name='my-cluster',
> > address='',
> > > connectorName='artemis', clientFailureCheckPeriod=30000,
> > > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > > maxRetryInterval=32000, initialConnectAttempts=20,
> reconnectAttempts=-1,
> > > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > > staticConnectors=[], discoveryGroupName='my-discovery-group',
> maxHops=1,
> > > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > > clusterNotificationAttempts=2}
> > >
> > > 2024-01-11 18:24:14,081 DEBUG
> > > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > > BackupManager connecting to DiscoveryBackupConnector
> > > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> > >
> > > 2024-01-11 18:24:14,082 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Starting Backup Server
> > >
> > > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Setting server state as started
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > looking up the node through nodeLocator.locateNode()
> > >
> > > 2024-01-11 18:24:14,083 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Connecting towards a possible live, connection
> > > information=Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> > >
> > > 2024-01-11 18:24:14,083 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Calling
> > >
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> > >
> > > java.lang.Exception:trace <ftp://java.lang.Exception/trace>
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0
> <ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0
> <ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0
> <ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0
> <ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0
> <ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0
> <ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4
> rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0
> <ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>
> (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0
> <ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > Trying reconnection attempt 0/0
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > Trying to connect with
> > >
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> > > <mailto:connectorFactory
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> > >
> > > and currentConnectorConfig: TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Connector NettyConnector
> > > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> > useServlet=false,
> > > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > > using native epoll
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Started EPOLL Netty Connector version 4.1.86.Final to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> <ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Remote destination:
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.25:61616 <ftp://10.44.0.25/61616>
> > >
> > >
> > > Thanks,
> > > Paul
> > >
> > > From: Justin Bertram <jb...@apache.org>
> > > Date: Wednesday, December 13, 2023 at 3:23 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > > failback
> > > ActiveMQ Artemis ships with an example in the
> > > examples/features/ha/replicated-failback directory that demonstrates
> how
> > > this works. I ran this example and it worked fine. I also started up
> the
> > > brokers manually and triggered failover and failback. Again, everything
> > > worked fine. My only guess is that there's something configured wrong
> or
> > > perhaps there's an environmental issue that's preventing failback from
> > > working properly for you. If you could provide clear steps on how to
> > > reproduce this issue I could investigate further.
> > >
> > >
> > > Justin
> > >
> > > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> > paul.shields@hpe.com
> > > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I am having an issue with failback to the master broker. I start the
> > > > master broker then the slave broker. I see the replication from the
> > > master
> > > > to the slave happen. I stop the master/active broker and a failover
> > > happens
> > > > to the slave as expected. When the master is started a failback to it
> > is
> > > > expected but does not happen. The following is my ha-policy config.
> > > Did I
> > > > miss something in the ha-policy configuration? We are using version
> > > 2.28.0
> > > > and java17.
> > > >
> > > > Thanks,
> > > > Paul
> > > >
> > > >
> > > > Master broker.xml config
> > > >
> > > > <ha-policy>
> > > >
> > > > <replication>
> > > >
> > > > <master>
> > > >
> > > > <check-for-live-server>true</check-for-live-server>
> > > >
> > > > </master>
> > > >
> > > > </replication>
> > > >
> > > > </ha-policy>
> > > >
> > > > Slave broker.xml config
> > > >
> > > > <ha-policy>
> > > >
> > > > <replication>
> > > >
> > > > <slave>
> > > >
> > > > <allow-failback>true</allow-failback>
> > > >
> > > > </slave>
> > > >
> > > > </replication>
> > > >
> > > > </ha-policy>
> > > >
> > > > The following are the logs from the master and slave brokers
> > > > HA “master” broker log after restart
> > > > 2023-12-08 20:42:57,509 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883
> <ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> > for
> > > > protocols [MQTT]
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > >
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> <ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> > for
> > > > protocols [CORE]
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221007: Server is now live
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > > Starting embedded web server
> > > > 2023-12-08 20:42:57,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 15, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,627 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3
> > > > 2023-12-08 20:42:57,657 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > > > [ID=47edf16f<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > > [ID=47edf16f>,
> > > > local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
> > > > 2023-12-08 20:42:57,720 INFO
> > > > [org.apache.activemq.hawtio.branding.PluginContextListener]
> Initialized
> > > > activemq-branding plugin
> > > > 2023-12-08 20:42:57,737 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is connected
> > > > 2023-12-08 20:42:57,755 INFO
> > > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > > artemis-plugin plugin
> > > > 2023-12-08 20:42:57,766 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 18, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,767 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 20, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > > Initialising
> > > > hawtio services
> > > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > > Configuration
> > > > will be discovered via system properties
> > > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > > Hawtio 2.15.0
> > > > 2023-12-08 20:42:57,975 INFO
> > > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > > authentication filter, JAAS realm: "activemq" authorized role(s):
> > "admin"
> > > > role principal classes:
> > > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > > servlet is disabled
> > > > 2023-12-08 20:42:57,989 INFO
> > > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia
> overridden
> > > > property: [key=policyLocation,
> > > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > > HTTP Server started at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> > >
> > > >
> > > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > > Artemis Jolokia REST API available at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> > >
> > > >
> > > > 2023-12-08
> > > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> > >
> > > >
> > > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > > Console
> > > > available at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> > >
> > > >
> > > > 2023-12-08
> > > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> > >
> > > >
> > > > 20:42:58,455 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl]
> > > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > > > [ID=f85ec51b<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > > [ID=f85ec51b>,
> > > > local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
> > > > 2023-12-08 20:42:59,524 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > > > [ID=7cef23a2<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > > [ID=7cef23a2>,
> > > > local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
> > > > 2023-12-08 20:42:59,535 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 22, result: NO_BINDINGS
> > > > 2023-12-08 20:42:59,536 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> > connectionID:
> > > > 7cef23a2
> > > > 2023-12-08 20:42:59,549 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 24, result: NO_BINDINGS
> > > > 2023-12-08 20:42:59,550 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> > connectionID:
> > > > 7cef23a2
> > > > 2023-12-08 20:42:59,583 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 26, result: OK
> > > > 2023-12-08 20:42:59,584 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=true]@5f46053b
> > > > 2023-12-08 20:42:59,593 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> > created
> > > > consumer with ID: 0, with session name:
> > > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > > 2023-12-08 20:42:59,594 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 28, result: OK
> > > > 2023-12-08 20:42:59,616 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > > 2023-12-08 20:42:59,615 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[
> *AMQ*RESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > > 2023-12-08 20:42:59,621 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 31, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,621 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[
> *AMQ*RoutingName=ExpiryQueue,*AMQ*Distance=0,*AMQ*Address=ExpiryQueue,
> *AMQ*NotifType=BINDING_ADDED,*AMQ*Binding_ID=7,*AMQ*
> FilterString=NULL-value,*AMQ*NotifTimestamp=1702068179613,*AMQ*ClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
> > > > with transaction: null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 32, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[
> *AMQ*RoutingName=DLQ,*AMQ*Distance=0,*AMQ*Address=DLQ,*AMQ*
> NotifType=BINDING_ADDED,*AMQ*Binding_ID=3,*AMQ*FilterString=NULL-value,
> *AMQ*NotifTimestamp=1702068179614,*AMQ*ClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
> > > > with transaction: null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 35, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[
> *AMQ*RESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction:
> null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 39, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:43:08,447 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > > > [ID=1d1d79ce<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > > [ID=1d1d79ce>,
> > > > local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
> > > > 2023-12-08 20:43:08,464 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > > > [ID=8f88c6d5<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > > [ID=8f88c6d5>,
> > > > local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
> > > >
> > > >
> > > > HA “slave” broker log.
> > > >
> > > > 2023-12-08 20:37:58,374 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0
> [null]
> > > > started, waiting live to fail before it gets active
> > > >
> > > > 2023-12-08 20:37:59,020 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > > synchronized with live server,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > > >
> > > > 2023-12-08 20:38:00,032 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221031: backup announced
> > > > 2023-12-08 20:42:27,415 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221084: Requested 0 quorum votes
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221071: Failing over based on quorum vote results.
> > > > 2023-12-08 20:42:27,431 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212037: Connection failure to
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > > 10.44.0.20:61616 <ftp://10.44.0.20/61616> has been detected:
> AMQ219015: The connection was
> > > > disconnected because of server shutdown [code=DISCONNECTED]
> > > > 2023-12-08 20:42:27,431 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212037: Connection failure to
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > > 10.44.0.20:61616 <ftp://10.44.0.20/61616> has been detected:
> AMQ219015: The connection was
> > > > disconnected because of server shutdown [code=DISCONNECTED]
> > > > 2023-12-08 20:42:27,434 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > > 2023-12-08 20:42:27,574 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > > 2023-12-08 20:42:27,575 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > > 2023-12-08 20:42:27,578 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > > 2023-12-08 20:42:27,578 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > > 2023-12-08 20:42:27,611 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221007: Server is now live
> > > > 2023-12-08 20:42:27,699 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883
> <ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> > for
> > > > protocols [MQTT]
> > > > 2023-12-08 20:42:27,717 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > >
> cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> <ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> > for
> > > > protocols [CORE]
> > > > 2023-12-08 20:42:51,946 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > > [ID=df357d0e<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > [ID=df357d0e>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > > > 2023-12-08 20:42:56,977 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > > destroyed
> > > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > > [ID=df357d0e<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > [ID=df357d0e>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > > > 2023-12-08 20:42:57,443 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212004: Failed to connect to server.
> > > > 2023-12-08 20:42:57,445 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 26, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,447 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6
> > > > 2023-12-08 20:42:57,512 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > > failure detected. Unblocking a blocking call that will never get a
> > > response
> > > > 2023-12-08 20:42:57,513 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is unable to connect to
> destination.
> > > > Retrying
> > > > 2023-12-08 20:42:57,629 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > > > [ID=483170f6<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > > [ID=483170f6>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
> > > > 2023-12-08 20:42:57,630 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > > > [ID=4514a2b3<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > > [ID=4514a2b3>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
> > > > 2023-12-08 20:42:57,648 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > > > [ID=fcde4d78<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > > [ID=fcde4d78>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
> > > > 2023-12-08 20:42:57,663 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 29, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,663 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> > connectionID:
> > > > fcde4d78
> > > > 2023-12-08 20:42:57,679 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 31, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,679 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> > connectionID:
> > > > fcde4d78
> > > > 2023-12-08 20:42:57,708 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 33, result: OK
> > > > 2023-12-08 20:42:57,709 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=true]@3645037f
> > > > 2023-12-08 20:42:57,719 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> > created
> > > > consumer with ID: 0, with session name:
> > > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > > 2023-12-08 20:42:57,720 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 35, result: OK
> > > > 2023-12-08 20:42:57,755 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > > 2023-12-08 20:42:57,753 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[
> *AMQ*RESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > > 2023-12-08 20:42:57,760 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 38, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,761 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[
> *AMQ*RoutingName=DLQ,*AMQ*Distance=0,*AMQ*Address=DLQ,*AMQ*
> NotifType=BINDING_ADDED,*AMQ*Binding_ID=3,*AMQ*FilterString=NULL-value,
> *AMQ*NotifTimestamp=1702068177753,*AMQ*ClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
> > > > with transaction: null
> > > > 2023-12-08 20:42:57,761 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 41, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[
> *AMQ*RoutingName=ExpiryQueue,*AMQ*Distance=0,*AMQ*Address=ExpiryQueue,
> *AMQ*NotifType=BINDING_ADDED,*AMQ*Binding_ID=7,*AMQ*
> FilterString=NULL-value,*AMQ*NotifTimestamp=1702068177753,*AMQ*ClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
> > > > with transaction: null
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 43, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[
> *AMQ*RESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction:
> null
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 44, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:59,607 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is connected
> > > > 2023-12-08 20:42:59,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 46, result: OK
> > > > 2023-12-08 20:42:59,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 48, result: OK
> > > > 2023-12-08 20:43:07,437 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > > failure detected. Unblocking a blocking call that will never get a
> > > response
> > > >
> > > >
> > > >
> > >
> >
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
RE: Trouble with Replication HA Master/Slave config performing failback
Posted by John Lilley <jo...@redpointglobal.com.INVALID>.
Hi Justin,
This does seem counter-intuitive. We’re using replication, not shared storage. So even though the persistent data will be stored pod-local, enabling persistence has the knock-on effect of also making it available to the backup server in addition to whatever the “replication” model does?
john
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>
Sent: Friday, January 19, 2024 9:42 AM
To: users@activemq.apache.org
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> ...do we currently set persistenceEnable=false? I would expect there’s no reason for setting it true, since the pod-local storage will evaporate with the pod.
You definitely want to enable persistence because only persistent data will be available on the backup.
Justin
On Fri, Jan 19, 2024 at 10:21 AM John Lilley <jo...@redpointglobal.com.invalid>> wrote:
Hi Paul,
Thanks for the information! It sounds like you have experience with AMQ in K8S. Can you share your two broker.xml files with us?
Lino: do we currently set persistenceEnable=false? I would expect there’s no reason for setting it true, since the pod-local storage will evaporate with the pod.
Thanks
John
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,oxNBEUK4DA5mcfKgwBsn18k6NiUeBq7tvknA26MQvjAU9Rp2rY9KLKCPT1AgeaGlF70DMTEJQQIgrmI57_j35NweP5IX_CT1KzurdACsfSU1vLrkUiPMR7U,&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Shields, Paul Michael <pa...@hpe.com>>
Sent: Friday, January 19, 2024 8:51 AM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hi John,
The <node-manager-lock-directory>. Needs to be specified in both the master and slave broker.xml files.
the nodeUUID which is contained in the server lock file was changing between active broker startups. The server.lock file is stored in the /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our current configuration of the broker has persistenceEnable=false which causes any data/broker state to be removed when the broker pod is destroyed. Thus a new nodeUUID was being generated on each Primary broker startup. When the Primary broker instance starts up it looks for an existing server.lock file. When it does not find one it generates a new nodeUUID and places it in the server.lock file, otherwise it uses the nodeUUID contained within the existing server.lock file. The Primary broker then communicates this nodeUUID through the cluster connection to the Standby broker instance on startup. The Standby broker instance uses the nodeUUID to determine if the Primary broker instance is part of its HA pair. The Standby broker instance saved off the original nodeUUID when the HA cluster was first started and since the recently communicated nodeUUID does not match the nodeUUID that it previously stored the fail back is not performed. Once persistence was enabled the Standby broker instance performed the fallback to the Primary instance and was no longer in the "active" state
This is what I have observed in our Kubernetes environment. Justin please make any corrections to my description of the nodeUUID use.
Paul
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Date: Thursday, January 18, 2024 at 5:49 PM
To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Justin,
Can you elaborate on the lock file in the <node-manager-lock-directory>?
Does it need to be specified in the broker.xml for both the master and the slave?
Does it actually get used as a “lock” in the concurrency sense, to coordinate the live/backup election?
Thanks
john
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,0rYuECbqGwpo9IjFKJrQQrfyzL5V_aEiqSXV25I7DeF4wuQn3I6z1sJCY1NkNJ-bjdpV8nVix_VDU0mK5tgrxuSQaSyrlVLxE5EiTf0Z8G4F&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Monday, January 15, 2024 9:56 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
In the replication case you still need a PV in K8s, but I don't believe you need to put the whole journal on the PV (and suffer the performance penalty). You should just need to point the node-manager-lock-directory in broker.xml to the PV.
Justin
On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
Based on your reply in the provided link, are you saying that even for the replication HA case, you still need an external FileShare PV, in K8s, so that the nodeID can be persisted between restarts. In this use case, is the performance of this FileShare less of a concern than it is for the shared-store use case?
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
-----Original Message-----
From: Justin Bertram <jb...@apache.org>>
Sent: Friday, January 12, 2024 4:53 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
I sent that email on December 15. You can find it on the web-based mailing
list interface [1].
Justin
[1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l
On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <pa...@hpe.com>>
wrote:
> Hi Justin,
>
> Forgive me but I don’t remember a journal persistence issue discussion. I
> did an email search too and did not see that discussion either. Would you
> please refresh me on the discussion? I may have inadvertently deleted that?
>
> I do see data being replicated to my production slave/standby instance on
> initial startup of the two broker instances. Failover works as expected.
> But when the primary instance starts backup I do not see the active slave
> replicate to the starting primary and I do not see the active slave restart
> and go to the backup server state.
>
> What is the communication sequence when the primary starts up after it
> quit with the active backup broker? In other words, how is the active
> backup broker notified to start replication to the newly started primary
> broker?
>
> I ran the replicated-failback example again with TRACE level logging
> turned on and I do see the attemptFailback=false message in the
> target/server1/log/artemis.log file at startup, so it must be changed later
> or am I not understanding the purpose of this flag?
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> discoveryInitialWaitTimeout=10000}]
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost, b=null],
> nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost)
>
> From my production slave broker log
>
>
> 024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> Thanks,
> Paul
> From: Justin Bertram <jb...@apache.org>>
> Date: Friday, January 12, 2024 at 3:38 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > Is it possible to use jgroups instead of UDP for the cluster connection
> in the example?
>
> Yes. Check out examples/features/clustered/clustered-jgroups for an example
> configuration. Keep in mind that UDP, JGroups, etc. is only used for
> initial discovery. Once brokers are discovered then normal TCP connections
> are created.
>
> Before I continue looking through your logs, etc. can you confirm that you
> resolved the journal persistence issue I described in my previous email? If
> that's not resolved there's no chance for failback to occur.
>
>
> Justin
>
>
>
> On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>>
> wrote:
>
> > Hi Justin,
> >
> > I tried the example and it works. I turned up logging to get a success
> > path through the code to maybe see where things get de-railed. Is it
> > possible to use jgroups instead of UDP for the cluster connection in the
> > example? That would give me a better success signature.
> >
> > My standby instance has this in the broker.xml which is the same as in
> the
> > example:
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > I am seeing this series of log messages that suggests that failback is
> not
> > enabled. Is the attemptFailback=false the default at this point in
> startup
> > and it is changed later or it there something gone wrong at this point?
> >
> > 2024-01-11 18:24:14,066 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > QuorumManager(server=null) about node up =
> > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> > <mailto:
<mailto:%0b>> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> >
> > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > ClientSessionFactoryImpl received backup update for live/backup pair =
> > TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > / null but it didn't belong to TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,067 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Cluster Connected
> >
> > 2024-01-11 18:24:14,068 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting backup manager
> >
> > 2024-01-11 18:24:14,069 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > backup config ClusterConnectionConfiguration{name='my-cluster',
> address='',
> > connectorName='artemis', clientFailureCheckPeriod=30000,
> > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > clusterNotificationAttempts=2}
> >
> > 2024-01-11 18:24:14,081 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > 2024-01-11 18:24:14,083 TRACE
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> >
> > java.lang.Exception:trace<ftp://java.lang.Exception/trace>
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4 rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying reconnection attempt 0/0
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18> > > <mailto:connectorFactory<mailto:connectorFactory>
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18>
> >
> > and currentConnectorConfig: TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Connector NettyConnector
> > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using native epoll
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Started EPOLL Netty Connector version 4.1.86.Final to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Remote destination:
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.25:61616<ftp://10.44.0.25/61616>
> >
> >
> > Thanks,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>>
> > Date: Wednesday, December 13, 2023 at 3:23 PM
> > To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > ActiveMQ Artemis ships with an example in the
> > examples/features/ha/replicated-failback directory that demonstrates how
> > this works. I ran this example and it worked fine. I also started up the
> > brokers manually and triggered failover and failback. Again, everything
> > worked fine. My only guess is that there's something configured wrong or
> > perhaps there's an environmental issue that's preventing failback from
> > working properly for you. If you could provide clear steps on how to
> > reproduce this issue I could investigate further.
> >
> >
> > Justin
> >
> > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am having an issue with failback to the master broker. I start the
> > > master broker then the slave broker. I see the replication from the
> > master
> > > to the slave happen. I stop the master/active broker and a failover
> > happens
> > > to the slave as expected. When the master is started a failback to it
> is
> > > expected but does not happen. The following is my ha-policy config.
> > Did I
> > > miss something in the ha-policy configuration? We are using version
> > 2.28.0
> > > and java17.
> > >
> > > Thanks,
> > > Paul
> > >
> > >
> > > Master broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <master>
> > >
> > > <check-for-live-server>true</check-for-live-server>
> > >
> > > </master>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > Slave broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > The following are the logs from the master and slave brokers
> > > HA “master” broker log after restart
> > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > Starting embedded web server
> > > 2023-12-08 20:42:57,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 15, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,627 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3
> > > 2023-12-08 20:42:57,657 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2> > > > [ID=47edf16f<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2>
> > [ID=47edf16f>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/10.44.0.18:34508<http://10.44.0.18:34508>]]
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > > activemq-branding plugin
> > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > artemis-plugin plugin
> > > 2023-12-08 20:42:57,766 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 18, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,767 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 20, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > Initialising
> > > hawtio services
> > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > Configuration
> > > will be discovered via system properties
> > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > Hawtio 2.15.0
> > > 2023-12-08 20:42:57,975 INFO
> > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > authentication filter, JAAS realm: "activemq" authorized role(s):
> "admin"
> > > role principal classes:
> > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > servlet is disabled
> > > 2023-12-08 20:42:57,989 INFO
> > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > > property: [key=policyLocation,
> > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > HTTP Server started at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > >
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > Artemis Jolokia REST API available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > >
> > >
> > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > Console
> > > available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> > >
> > > 20:42:58,455 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl]
> > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d> > > > [ID=f85ec51b<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d>
> > [ID=f85ec51b>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:49865<http://127.0.0.6:49865>]]
> > > 2023-12-08 20:42:59,524 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858> > > > [ID=7cef23a2<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858>
> > [ID=7cef23a2>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:59691<http://127.0.0.6:59691>]]
> > > 2023-12-08 20:42:59,535 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 22, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,536 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,549 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 24, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,550 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,583 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: OK
> > > 2023-12-08 20:42:59,584 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@5f46053b
> > > 2023-12-08 20:42:59,593 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > 2023-12-08 20:42:59,594 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 28, result: OK
> > > 2023-12-08 20:42:59,616 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:59,615 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 31, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 32, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 35, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 39, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:43:08,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256> > > > [ID=1d1d79ce<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256>
> > [ID=1d1d79ce>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:60787<http://127.0.0.6:60787>]]
> > > 2023-12-08 20:43:08,464 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7> > > > [ID=8f88c6d5<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7>
> > [ID=8f88c6d5>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:40065<http://127.0.0.6:40065>]]
> > >
> > >
> > > HA “slave” broker log.
> > >
> > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > synchronized with live server,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > >
> > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221031: backup announced
> > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221084: Requested 0 quorum votes
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221071: Failing over based on quorum vote results.
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:51,946 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:56,977 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > destroyed
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2023-12-08 20:42:57,445 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6
> > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > > Retrying
> > > 2023-12-08 20:42:57,629 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > > [ID=483170f6<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:55911<http://127.0.0.6:55911>]]
> > > 2023-12-08 20:42:57,630 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > > [ID=4514a2b3<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:60199<http://127.0.0.6:60199>]]
> > > 2023-12-08 20:42:57,648 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > > [ID=fcde4d78<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:53221<http://127.0.0.6:53221>]]
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 29, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 31, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,708 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 33, result: OK
> > > 2023-12-08 20:42:57,709 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@3645037f
> > > 2023-12-08 20:42:57,719 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 35, result: OK
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:57,753 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > 2023-12-08 20:42:57,760 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 38, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, > > > with transaction: null
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 41, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 43, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 44, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 46, result: OK
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 48, result: OK
> > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > >
> > >
> > >
> >
>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by Justin Bertram <jb...@apache.org>.
> ...do we currently set persistenceEnable=false? I would expect there’s
no reason for setting it true, since the pod-local storage will evaporate
with the pod.
You definitely want to enable persistence because only persistent data will
be available on the backup.
Justin
On Fri, Jan 19, 2024 at 10:21 AM John Lilley
<jo...@redpointglobal.com.invalid> wrote:
> Hi Paul,
>
>
>
> Thanks for the information! It sounds like you have experience with AMQ
> in K8S. Can you share your two broker.xml files with us?
>
>
>
> Lino: do we currently set persistenceEnable=false? I would expect
> there’s no reason for setting it true, since the pod-local storage will
> evaporate with the pod.
>
>
>
> Thanks
>
> John
>
>
>
>
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> *From:* Shields, Paul Michael <pa...@hpe.com>
> *Sent:* Friday, January 19, 2024 8:51 AM
> *To:* users@activemq.apache.org
> *Subject:* Re: Trouble with Replication HA Master/Slave config performing
> failback
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Hi John,
>
>
>
> The <node-manager-lock-directory>. Needs to be specified in both the
> master and slave broker.xml files.
>
>
>
> the nodeUUID which is contained in the server lock file was changing
> between active broker startups. The server.lock file is stored in the
> /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our
> current configuration of the broker has persistenceEnable=false which
> causes any data/broker state to be removed when the broker pod is
> destroyed. Thus a new nodeUUID was being generated on each Primary broker
> startup. When the Primary broker instance starts up it looks for an
> existing server.lock file. When it does not find one it generates a new
> nodeUUID and places it in the server.lock file, otherwise it uses the
> nodeUUID contained within the existing server.lock file. The Primary broker
> then communicates this nodeUUID through the cluster connection to the
> Standby broker instance on startup. The Standby broker instance uses the
> nodeUUID to determine if the Primary broker instance is part of its HA
> pair. The Standby broker instance saved off the original nodeUUID when the
> HA cluster was first started and since the recently communicated nodeUUID
> does not match the nodeUUID that it previously stored the fail back is not
> performed. Once persistence was enabled the Standby broker instance
> performed the fallback to the Primary instance and was no longer in the
> "active" state
>
>
>
> This is what I have observed in our Kubernetes environment. Justin please
> make any corrections to my description of the nodeUUID use.
>
>
>
> Paul
>
> *From: *John Lilley <jo...@redpointglobal.com.INVALID>
> *Date: *Thursday, January 18, 2024 at 5:49 PM
> *To: *users@activemq.apache.org <us...@activemq.apache.org>
> *Subject: *RE: Trouble with Replication HA Master/Slave config performing
> failback
>
> Hi Justin,
>
>
>
> Can you elaborate on the lock file in the <node-manager-lock-directory>?
>
> Does it need to be specified in the broker.xml for both the master and the
> slave?
>
> Does it actually get used as a “lock” in the concurrency sense, to
> coordinate the live/backup election?
>
>
>
> Thanks
>
> john
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,0rYuECbqGwpo9IjFKJrQQrfyzL5V_aEiqSXV25I7DeF4wuQn3I6z1sJCY1NkNJ-bjdpV8nVix_VDU0mK5tgrxuSQaSyrlVLxE5EiTf0Z8G4F&typo=1>
>
> *John Lilley *
>
> *Data Management Chief Architect, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Monday, January 15, 2024 9:56 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Trouble with Replication HA Master/Slave config performing
> failback
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> In the replication case you still need a PV in K8s, but I don't believe
> you need to put the whole journal on the PV (and suffer the performance
> penalty). You should just need to point the node-manager-lock-directory in
> broker.xml to the PV.
>
>
>
>
>
> Justin
>
>
>
> On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hi Justin,
>
> Based on your reply in the provided link, are you saying that even for the
> replication HA case, you still need an external FileShare PV, in K8s, so
> that the nodeID can be persisted between restarts. In this use case, is the
> performance of this FileShare less of a concern than it is for the
> shared-store use case?
>
> Thanks,
> Lino
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Friday, January 12, 2024 4:53 PM
> To: users@activemq.apache.org
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
>
> *** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ***
>
> I sent that email on December 15. You can find it on the web-based mailing
> list interface [1].
>
> Justin
>
> [1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l
>
> On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <
> paul.shields@hpe.com>
> wrote:
>
> > Hi Justin,
> >
> > Forgive me but I don’t remember a journal persistence issue discussion. I
> > did an email search too and did not see that discussion either. Would you
> > please refresh me on the discussion? I may have inadvertently deleted
> that?
> >
> > I do see data being replicated to my production slave/standby instance on
> > initial startup of the two broker instances. Failover works as expected.
> > But when the primary instance starts backup I do not see the active slave
> > replicate to the starting primary and I do not see the active slave
> restart
> > and go to the backup server state.
> >
> > What is the communication sequence when the primary starts up after it
> > quit with the active backup broker? In other words, how is the active
> > backup broker notified to start replication to the newly started primary
> > broker?
> >
> > I ran the replicated-failback example again with TRACE level logging
> > turned on and I do see the attemptFailback=false message in the
> > target/server1/log/artemis.log file at startup, so it must be changed
> later
> > or am I not understanding the purpose of this flag?
> >
> > 2024-01-12 14:40:18,341 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> > discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=localhost, b=null],
> > nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=localhost)
> >
> > From my production slave broker log
> >
> >
> > 024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > Thanks,
> > Paul
> > From: Justin Bertram <jb...@apache.org>
> > Date: Friday, January 12, 2024 at 3:38 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > > Is it possible to use jgroups instead of UDP for the cluster connection
> > in the example?
> >
> > Yes. Check out examples/features/clustered/clustered-jgroups for an
> example
> > configuration. Keep in mind that UDP, JGroups, etc. is *only* used for
> > initial discovery. Once brokers are discovered then normal TCP
> connections
> > are created.
> >
> > Before I continue looking through your logs, etc. can you confirm that
> you
> > resolved the journal persistence issue I described in my previous email?
> If
> > that's not resolved there's no chance for failback to occur.
> >
> >
> > Justin
> >
> >
> >
> > On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> > paul.shields@hpe.com>
> > wrote:
> >
> > > Hi Justin,
> > >
> > > I tried the example and it works. I turned up logging to get a success
> > > path through the code to maybe see where things get de-railed. Is it
> > > possible to use jgroups instead of UDP for the cluster connection in
> the
> > > example? That would give me a better success signature.
> > >
> > > My standby instance has this in the broker.xml which is the same as in
> > the
> > > example:
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > I am seeing this series of log messages that suggests that failback is
> > not
> > > enabled. Is the attemptFailback=false the default at this point in
> > startup
> > > and it is changed later or it there something gone wrong at this point?
> > >
> > > 2024-01-11 18:24:14,066 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.Topology]
> Topology@7e0a913c
> > [owner=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > >
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > > QuorumManager(server=null) about node up =
> > > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > > Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null]
> > >
> > > 2024-01-11 18:24:14,067 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.Topology]
> Topology@7e0a913c
> > [owner=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > >
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > >
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> > > <mailto:
> <%0b>> >
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> > >
> > > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > > Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null]
> > >
> > > 2024-01-11 18:24:14,067 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > ClientSessionFactoryImpl received backup update for live/backup pair =
> > > TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > > / null but it didn't belong to TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > >
> > > 2024-01-11 18:24:14,067 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Cluster Connected
> > >
> > > 2024-01-11 18:24:14,068 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Starting backup manager
> > >
> > > 2024-01-11 18:24:14,069 DEBUG
> > > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > > backup config ClusterConnectionConfiguration{name='my-cluster',
> > address='',
> > > connectorName='artemis', clientFailureCheckPeriod=30000,
> > > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > > maxRetryInterval=32000, initialConnectAttempts=20,
> reconnectAttempts=-1,
> > > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > > staticConnectors=[], discoveryGroupName='my-discovery-group',
> maxHops=1,
> > > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > > clusterNotificationAttempts=2}
> > >
> > > 2024-01-11 18:24:14,081 DEBUG
> > > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > > BackupManager connecting to DiscoveryBackupConnector
> > > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> > >
> > > 2024-01-11 18:24:14,082 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Starting Backup Server
> > >
> > > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Setting server state as started
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > looking up the node through nodeLocator.locateNode()
> > >
> > > 2024-01-11 18:24:14,083 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Connecting towards a possible live, connection
> > > information=Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> > >
> > > 2024-01-11 18:24:14,083 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Calling
> > >
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> > >
> > > java.lang.Exception:trace <ftp://java.lang.Exception/trace>
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0
> <ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0
> <ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0
> <ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0
> <ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0
> <ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0
> <ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4
> rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0
> <ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>
> (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0
> <ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > Trying reconnection attempt 0/0
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > Trying to connect with
> > >
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> > > <mailto:connectorFactory
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> > >
> > > and currentConnectorConfig: TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Connector NettyConnector
> > > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> > useServlet=false,
> > > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > > using native epoll
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Started EPOLL Netty Connector version 4.1.86.Final to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> <ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Remote destination:
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.25:61616 <ftp://10.44.0.25/61616>
> > >
> > >
> > > Thanks,
> > > Paul
> > >
> > > From: Justin Bertram <jb...@apache.org>
> > > Date: Wednesday, December 13, 2023 at 3:23 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > > failback
> > > ActiveMQ Artemis ships with an example in the
> > > examples/features/ha/replicated-failback directory that demonstrates
> how
> > > this works. I ran this example and it worked fine. I also started up
> the
> > > brokers manually and triggered failover and failback. Again, everything
> > > worked fine. My only guess is that there's something configured wrong
> or
> > > perhaps there's an environmental issue that's preventing failback from
> > > working properly for you. If you could provide clear steps on how to
> > > reproduce this issue I could investigate further.
> > >
> > >
> > > Justin
> > >
> > > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> > paul.shields@hpe.com
> > > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I am having an issue with failback to the master broker. I start the
> > > > master broker then the slave broker. I see the replication from the
> > > master
> > > > to the slave happen. I stop the master/active broker and a failover
> > > happens
> > > > to the slave as expected. When the master is started a failback to it
> > is
> > > > expected but does not happen. The following is my ha-policy config.
> > > Did I
> > > > miss something in the ha-policy configuration? We are using version
> > > 2.28.0
> > > > and java17.
> > > >
> > > > Thanks,
> > > > Paul
> > > >
> > > >
> > > > Master broker.xml config
> > > >
> > > > <ha-policy>
> > > >
> > > > <replication>
> > > >
> > > > <master>
> > > >
> > > > <check-for-live-server>true</check-for-live-server>
> > > >
> > > > </master>
> > > >
> > > > </replication>
> > > >
> > > > </ha-policy>
> > > >
> > > > Slave broker.xml config
> > > >
> > > > <ha-policy>
> > > >
> > > > <replication>
> > > >
> > > > <slave>
> > > >
> > > > <allow-failback>true</allow-failback>
> > > >
> > > > </slave>
> > > >
> > > > </replication>
> > > >
> > > > </ha-policy>
> > > >
> > > > The following are the logs from the master and slave brokers
> > > > HA “master” broker log after restart
> > > > 2023-12-08 20:42:57,509 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883
> <ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> > for
> > > > protocols [MQTT]
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > >
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> <ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> > for
> > > > protocols [CORE]
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221007: Server is now live
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > > Starting embedded web server
> > > > 2023-12-08 20:42:57,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 15, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,627 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3
> > > > 2023-12-08 20:42:57,657 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > > > [ID=47edf16f<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > > [ID=47edf16f>,
> > > > local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
> > > > 2023-12-08 20:42:57,720 INFO
> > > > [org.apache.activemq.hawtio.branding.PluginContextListener]
> Initialized
> > > > activemq-branding plugin
> > > > 2023-12-08 20:42:57,737 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is connected
> > > > 2023-12-08 20:42:57,755 INFO
> > > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > > artemis-plugin plugin
> > > > 2023-12-08 20:42:57,766 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 18, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,767 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 20, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > > Initialising
> > > > hawtio services
> > > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > > Configuration
> > > > will be discovered via system properties
> > > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > > Hawtio 2.15.0
> > > > 2023-12-08 20:42:57,975 INFO
> > > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > > authentication filter, JAAS realm: "activemq" authorized role(s):
> > "admin"
> > > > role principal classes:
> > > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > > servlet is disabled
> > > > 2023-12-08 20:42:57,989 INFO
> > > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia
> overridden
> > > > property: [key=policyLocation,
> > > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > > HTTP Server started at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> > >
> > > >
> > > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > > Artemis Jolokia REST API available at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> > >
> > > >
> > > > 2023-12-08
> > > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> > >
> > > >
> > > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > > Console
> > > > available at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> > >
> > > >
> > > > 2023-12-08
> > > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> > >
> > > >
> > > > 20:42:58,455 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl]
> > > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > > > [ID=f85ec51b<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > > [ID=f85ec51b>,
> > > > local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
> > > > 2023-12-08 20:42:59,524 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > > > [ID=7cef23a2<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > > [ID=7cef23a2>,
> > > > local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
> > > > 2023-12-08 20:42:59,535 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 22, result: NO_BINDINGS
> > > > 2023-12-08 20:42:59,536 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> > connectionID:
> > > > 7cef23a2
> > > > 2023-12-08 20:42:59,549 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 24, result: NO_BINDINGS
> > > > 2023-12-08 20:42:59,550 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> > connectionID:
> > > > 7cef23a2
> > > > 2023-12-08 20:42:59,583 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 26, result: OK
> > > > 2023-12-08 20:42:59,584 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=true]@5f46053b
> > > > 2023-12-08 20:42:59,593 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> > created
> > > > consumer with ID: 0, with session name:
> > > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > > 2023-12-08 20:42:59,594 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 28, result: OK
> > > > 2023-12-08 20:42:59,616 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > > 2023-12-08 20:42:59,615 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[
> *AMQ*RESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > > 2023-12-08 20:42:59,621 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 31, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,621 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[
> *AMQ*RoutingName=ExpiryQueue,*AMQ*Distance=0,*AMQ*Address=ExpiryQueue,
> *AMQ*NotifType=BINDING_ADDED,*AMQ*Binding_ID=7,*AMQ*
> FilterString=NULL-value,*AMQ*NotifTimestamp=1702068179613,*AMQ*ClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
> > > > with transaction: null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 32, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[
> *AMQ*RoutingName=DLQ,*AMQ*Distance=0,*AMQ*Address=DLQ,*AMQ*
> NotifType=BINDING_ADDED,*AMQ*Binding_ID=3,*AMQ*FilterString=NULL-value,
> *AMQ*NotifTimestamp=1702068179614,*AMQ*ClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
> > > > with transaction: null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 35, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[
> *AMQ*RESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction:
> null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 39, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:43:08,447 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > > > [ID=1d1d79ce<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > > [ID=1d1d79ce>,
> > > > local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
> > > > 2023-12-08 20:43:08,464 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > > > [ID=8f88c6d5<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > > [ID=8f88c6d5>,
> > > > local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
> > > >
> > > >
> > > > HA “slave” broker log.
> > > >
> > > > 2023-12-08 20:37:58,374 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0
> [null]
> > > > started, waiting live to fail before it gets active
> > > >
> > > > 2023-12-08 20:37:59,020 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > > synchronized with live server,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > > >
> > > > 2023-12-08 20:38:00,032 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221031: backup announced
> > > > 2023-12-08 20:42:27,415 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221084: Requested 0 quorum votes
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221071: Failing over based on quorum vote results.
> > > > 2023-12-08 20:42:27,431 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212037: Connection failure to
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > > 10.44.0.20:61616 <ftp://10.44.0.20/61616> has been detected:
> AMQ219015: The connection was
> > > > disconnected because of server shutdown [code=DISCONNECTED]
> > > > 2023-12-08 20:42:27,431 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212037: Connection failure to
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > > 10.44.0.20:61616 <ftp://10.44.0.20/61616> has been detected:
> AMQ219015: The connection was
> > > > disconnected because of server shutdown [code=DISCONNECTED]
> > > > 2023-12-08 20:42:27,434 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > > 2023-12-08 20:42:27,574 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > > 2023-12-08 20:42:27,575 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > > 2023-12-08 20:42:27,578 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > > 2023-12-08 20:42:27,578 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > > 2023-12-08 20:42:27,611 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221007: Server is now live
> > > > 2023-12-08 20:42:27,699 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883
> <ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> > for
> > > > protocols [MQTT]
> > > > 2023-12-08 20:42:27,717 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > >
> cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> <ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> > for
> > > > protocols [CORE]
> > > > 2023-12-08 20:42:51,946 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > > [ID=df357d0e<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > [ID=df357d0e>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > > > 2023-12-08 20:42:56,977 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > > destroyed
> > > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > > [ID=df357d0e<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > [ID=df357d0e>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > > > 2023-12-08 20:42:57,443 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212004: Failed to connect to server.
> > > > 2023-12-08 20:42:57,445 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 26, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,447 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6
> > > > 2023-12-08 20:42:57,512 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > > failure detected. Unblocking a blocking call that will never get a
> > > response
> > > > 2023-12-08 20:42:57,513 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is unable to connect to
> destination.
> > > > Retrying
> > > > 2023-12-08 20:42:57,629 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > > > [ID=483170f6<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > > [ID=483170f6>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
> > > > 2023-12-08 20:42:57,630 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > > > [ID=4514a2b3<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > > [ID=4514a2b3>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
> > > > 2023-12-08 20:42:57,648 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > > > [ID=fcde4d78<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > > [ID=fcde4d78>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
> > > > 2023-12-08 20:42:57,663 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 29, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,663 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> > connectionID:
> > > > fcde4d78
> > > > 2023-12-08 20:42:57,679 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 31, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,679 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> > connectionID:
> > > > fcde4d78
> > > > 2023-12-08 20:42:57,708 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 33, result: OK
> > > > 2023-12-08 20:42:57,709 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=true]@3645037f
> > > > 2023-12-08 20:42:57,719 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> > created
> > > > consumer with ID: 0, with session name:
> > > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > > 2023-12-08 20:42:57,720 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 35, result: OK
> > > > 2023-12-08 20:42:57,755 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > > 2023-12-08 20:42:57,753 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[
> *AMQ*RESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > > 2023-12-08 20:42:57,760 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 38, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,761 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[
> *AMQ*RoutingName=DLQ,*AMQ*Distance=0,*AMQ*Address=DLQ,*AMQ*
> NotifType=BINDING_ADDED,*AMQ*Binding_ID=3,*AMQ*FilterString=NULL-value,
> *AMQ*NotifTimestamp=1702068177753,*AMQ*ClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
> > > > with transaction: null
> > > > 2023-12-08 20:42:57,761 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 41, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[
> *AMQ*RoutingName=ExpiryQueue,*AMQ*Distance=0,*AMQ*Address=ExpiryQueue,
> *AMQ*NotifType=BINDING_ADDED,*AMQ*Binding_ID=7,*AMQ*
> FilterString=NULL-value,*AMQ*NotifTimestamp=1702068177753,*AMQ*ClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
> > > > with transaction: null
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 43, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[
> *AMQ*RESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction:
> null
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 44, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:59,607 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is connected
> > > > 2023-12-08 20:42:59,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 46, result: OK
> > > > 2023-12-08 20:42:59,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 48, result: OK
> > > > 2023-12-08 20:43:07,437 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > > failure detected. Unblocking a blocking call that will never get a
> > > response
> > > >
> > > >
> > > >
> > >
> >
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
RE: Trouble with Replication HA Master/Slave config performing failback
Posted by John Lilley <jo...@redpointglobal.com.INVALID>.
Hi Paul,
Thanks for the information! It sounds like you have experience with AMQ in K8S. Can you share your two broker.xml files with us?
Lino: do we currently set persistenceEnable=false? I would expect there’s no reason for setting it true, since the pod-local storage will evaporate with the pod.
Thanks
John
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Shields, Paul Michael <pa...@hpe.com>
Sent: Friday, January 19, 2024 8:51 AM
To: users@activemq.apache.org
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hi John,
The <node-manager-lock-directory>. Needs to be specified in both the master and slave broker.xml files.
the nodeUUID which is contained in the server lock file was changing between active broker startups. The server.lock file is stored in the /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our current configuration of the broker has persistenceEnable=false which causes any data/broker state to be removed when the broker pod is destroyed. Thus a new nodeUUID was being generated on each Primary broker startup. When the Primary broker instance starts up it looks for an existing server.lock file. When it does not find one it generates a new nodeUUID and places it in the server.lock file, otherwise it uses the nodeUUID contained within the existing server.lock file. The Primary broker then communicates this nodeUUID through the cluster connection to the Standby broker instance on startup. The Standby broker instance uses the nodeUUID to determine if the Primary broker instance is part of its HA pair. The Standby broker instance saved off the original nodeUUID when the HA cluster was first started and since the recently communicated nodeUUID does not match the nodeUUID that it previously stored the fail back is not performed. Once persistence was enabled the Standby broker instance performed the fallback to the Primary instance and was no longer in the "active" state
This is what I have observed in our Kubernetes environment. Justin please make any corrections to my description of the nodeUUID use.
Paul
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Date: Thursday, January 18, 2024 at 5:49 PM
To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Justin,
Can you elaborate on the lock file in the <node-manager-lock-directory>?
Does it need to be specified in the broker.xml for both the master and the slave?
Does it actually get used as a “lock” in the concurrency sense, to coordinate the live/backup election?
Thanks
john
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,0rYuECbqGwpo9IjFKJrQQrfyzL5V_aEiqSXV25I7DeF4wuQn3I6z1sJCY1NkNJ-bjdpV8nVix_VDU0mK5tgrxuSQaSyrlVLxE5EiTf0Z8G4F&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Monday, January 15, 2024 9:56 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
In the replication case you still need a PV in K8s, but I don't believe you need to put the whole journal on the PV (and suffer the performance penalty). You should just need to point the node-manager-lock-directory in broker.xml to the PV.
Justin
On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
Based on your reply in the provided link, are you saying that even for the replication HA case, you still need an external FileShare PV, in K8s, so that the nodeID can be persisted between restarts. In this use case, is the performance of this FileShare less of a concern than it is for the shared-store use case?
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
-----Original Message-----
From: Justin Bertram <jb...@apache.org>>
Sent: Friday, January 12, 2024 4:53 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
I sent that email on December 15. You can find it on the web-based mailing
list interface [1].
Justin
[1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l
On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <pa...@hpe.com>>
wrote:
> Hi Justin,
>
> Forgive me but I don’t remember a journal persistence issue discussion. I
> did an email search too and did not see that discussion either. Would you
> please refresh me on the discussion? I may have inadvertently deleted that?
>
> I do see data being replicated to my production slave/standby instance on
> initial startup of the two broker instances. Failover works as expected.
> But when the primary instance starts backup I do not see the active slave
> replicate to the starting primary and I do not see the active slave restart
> and go to the backup server state.
>
> What is the communication sequence when the primary starts up after it
> quit with the active backup broker? In other words, how is the active
> backup broker notified to start replication to the newly started primary
> broker?
>
> I ran the replicated-failback example again with TRACE level logging
> turned on and I do see the attemptFailback=false message in the
> target/server1/log/artemis.log file at startup, so it must be changed later
> or am I not understanding the purpose of this flag?
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> discoveryInitialWaitTimeout=10000}]
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost, b=null],
> nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost)
>
> From my production slave broker log
>
>
> 024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> Thanks,
> Paul
> From: Justin Bertram <jb...@apache.org>>
> Date: Friday, January 12, 2024 at 3:38 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > Is it possible to use jgroups instead of UDP for the cluster connection
> in the example?
>
> Yes. Check out examples/features/clustered/clustered-jgroups for an example
> configuration. Keep in mind that UDP, JGroups, etc. is only used for
> initial discovery. Once brokers are discovered then normal TCP connections
> are created.
>
> Before I continue looking through your logs, etc. can you confirm that you
> resolved the journal persistence issue I described in my previous email? If
> that's not resolved there's no chance for failback to occur.
>
>
> Justin
>
>
>
> On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>>
> wrote:
>
> > Hi Justin,
> >
> > I tried the example and it works. I turned up logging to get a success
> > path through the code to maybe see where things get de-railed. Is it
> > possible to use jgroups instead of UDP for the cluster connection in the
> > example? That would give me a better success signature.
> >
> > My standby instance has this in the broker.xml which is the same as in
> the
> > example:
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > I am seeing this series of log messages that suggests that failback is
> not
> > enabled. Is the attemptFailback=false the default at this point in
> startup
> > and it is changed later or it there something gone wrong at this point?
> >
> > 2024-01-11 18:24:14,066 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > QuorumManager(server=null) about node up =
> > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> > <mailto:
<mailto:%0b>> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> >
> > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > ClientSessionFactoryImpl received backup update for live/backup pair =
> > TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > / null but it didn't belong to TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,067 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Cluster Connected
> >
> > 2024-01-11 18:24:14,068 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting backup manager
> >
> > 2024-01-11 18:24:14,069 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > backup config ClusterConnectionConfiguration{name='my-cluster',
> address='',
> > connectorName='artemis', clientFailureCheckPeriod=30000,
> > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > clusterNotificationAttempts=2}
> >
> > 2024-01-11 18:24:14,081 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > 2024-01-11 18:24:14,083 TRACE
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> >
> > java.lang.Exception:trace<ftp://java.lang.Exception/trace>
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4 rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying reconnection attempt 0/0
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18> > > <mailto:connectorFactory<mailto:connectorFactory>
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18>
> >
> > and currentConnectorConfig: TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Connector NettyConnector
> > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using native epoll
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Started EPOLL Netty Connector version 4.1.86.Final to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Remote destination:
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.25:61616<ftp://10.44.0.25/61616>
> >
> >
> > Thanks,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>>
> > Date: Wednesday, December 13, 2023 at 3:23 PM
> > To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > ActiveMQ Artemis ships with an example in the
> > examples/features/ha/replicated-failback directory that demonstrates how
> > this works. I ran this example and it worked fine. I also started up the
> > brokers manually and triggered failover and failback. Again, everything
> > worked fine. My only guess is that there's something configured wrong or
> > perhaps there's an environmental issue that's preventing failback from
> > working properly for you. If you could provide clear steps on how to
> > reproduce this issue I could investigate further.
> >
> >
> > Justin
> >
> > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am having an issue with failback to the master broker. I start the
> > > master broker then the slave broker. I see the replication from the
> > master
> > > to the slave happen. I stop the master/active broker and a failover
> > happens
> > > to the slave as expected. When the master is started a failback to it
> is
> > > expected but does not happen. The following is my ha-policy config.
> > Did I
> > > miss something in the ha-policy configuration? We are using version
> > 2.28.0
> > > and java17.
> > >
> > > Thanks,
> > > Paul
> > >
> > >
> > > Master broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <master>
> > >
> > > <check-for-live-server>true</check-for-live-server>
> > >
> > > </master>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > Slave broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > The following are the logs from the master and slave brokers
> > > HA “master” broker log after restart
> > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > Starting embedded web server
> > > 2023-12-08 20:42:57,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 15, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,627 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3
> > > 2023-12-08 20:42:57,657 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2> > > > [ID=47edf16f<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2>
> > [ID=47edf16f>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/10.44.0.18:34508<http://10.44.0.18:34508>]]
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > > activemq-branding plugin
> > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > artemis-plugin plugin
> > > 2023-12-08 20:42:57,766 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 18, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,767 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 20, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > Initialising
> > > hawtio services
> > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > Configuration
> > > will be discovered via system properties
> > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > Hawtio 2.15.0
> > > 2023-12-08 20:42:57,975 INFO
> > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > authentication filter, JAAS realm: "activemq" authorized role(s):
> "admin"
> > > role principal classes:
> > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > servlet is disabled
> > > 2023-12-08 20:42:57,989 INFO
> > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > > property: [key=policyLocation,
> > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > HTTP Server started at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > >
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > Artemis Jolokia REST API available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > >
> > >
> > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > Console
> > > available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> > >
> > > 20:42:58,455 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl]
> > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d> > > > [ID=f85ec51b<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d>
> > [ID=f85ec51b>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:49865<http://127.0.0.6:49865>]]
> > > 2023-12-08 20:42:59,524 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858> > > > [ID=7cef23a2<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858>
> > [ID=7cef23a2>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:59691<http://127.0.0.6:59691>]]
> > > 2023-12-08 20:42:59,535 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 22, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,536 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,549 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 24, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,550 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,583 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: OK
> > > 2023-12-08 20:42:59,584 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@5f46053b
> > > 2023-12-08 20:42:59,593 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > 2023-12-08 20:42:59,594 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 28, result: OK
> > > 2023-12-08 20:42:59,616 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:59,615 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 31, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 32, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 35, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 39, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:43:08,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256> > > > [ID=1d1d79ce<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256>
> > [ID=1d1d79ce>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:60787<http://127.0.0.6:60787>]]
> > > 2023-12-08 20:43:08,464 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7> > > > [ID=8f88c6d5<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7>
> > [ID=8f88c6d5>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:40065<http://127.0.0.6:40065>]]
> > >
> > >
> > > HA “slave” broker log.
> > >
> > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > synchronized with live server,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > >
> > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221031: backup announced
> > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221084: Requested 0 quorum votes
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221071: Failing over based on quorum vote results.
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:51,946 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:56,977 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > destroyed
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2023-12-08 20:42:57,445 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6
> > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > > Retrying
> > > 2023-12-08 20:42:57,629 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > > [ID=483170f6<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:55911<http://127.0.0.6:55911>]]
> > > 2023-12-08 20:42:57,630 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > > [ID=4514a2b3<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:60199<http://127.0.0.6:60199>]]
> > > 2023-12-08 20:42:57,648 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > > [ID=fcde4d78<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:53221<http://127.0.0.6:53221>]]
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 29, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 31, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,708 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 33, result: OK
> > > 2023-12-08 20:42:57,709 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@3645037f
> > > 2023-12-08 20:42:57,719 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 35, result: OK
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:57,753 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > 2023-12-08 20:42:57,760 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 38, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, > > > with transaction: null
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 41, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 43, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 44, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 46, result: OK
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 48, result: OK
> > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > >
> > >
> > >
> >
>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by "Shields, Paul Michael" <pa...@hpe.com>.
Hi John,
The <node-manager-lock-directory>. Needs to be specified in both the master and slave broker.xml files.
the nodeUUID which is contained in the server lock file was changing between active broker startups. The server.lock file is stored in the /home/jboss/amq-broker/data/journal directory in the pod filesystem. Our current configuration of the broker has persistenceEnable=false which causes any data/broker state to be removed when the broker pod is destroyed. Thus a new nodeUUID was being generated on each Primary broker startup. When the Primary broker instance starts up it looks for an existing server.lock file. When it does not find one it generates a new nodeUUID and places it in the server.lock file, otherwise it uses the nodeUUID contained within the existing server.lock file. The Primary broker then communicates this nodeUUID through the cluster connection to the Standby broker instance on startup. The Standby broker instance uses the nodeUUID to determine if the Primary broker instance is part of its HA pair. The Standby broker instance saved off the original nodeUUID when the HA cluster was first started and since the recently communicated nodeUUID does not match the nodeUUID that it previously stored the fail back is not performed. Once persistence was enabled the Standby broker instance performed the fallback to the Primary instance and was no longer in the "active" state
This is what I have observed in our Kubernetes environment. Justin please make any corrections to my description of the nodeUUID use.
Paul
From: John Lilley <jo...@redpointglobal.com.INVALID>
Date: Thursday, January 18, 2024 at 5:49 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: RE: Trouble with Replication HA Master/Slave config performing failback
Hi Justin,
Can you elaborate on the lock file in the <node-manager-lock-directory>?
Does it need to be specified in the broker.xml for both the master and the slave?
Does it actually get used as a “lock” in the concurrency sense, to coordinate the live/backup election?
Thanks
john
[rg]<https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>
Sent: Monday, January 15, 2024 9:56 PM
To: users@activemq.apache.org
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
In the replication case you still need a PV in K8s, but I don't believe you need to put the whole journal on the PV (and suffer the performance penalty). You should just need to point the node-manager-lock-directory in broker.xml to the PV.
Justin
On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
Based on your reply in the provided link, are you saying that even for the replication HA case, you still need an external FileShare PV, in K8s, so that the nodeID can be persisted between restarts. In this use case, is the performance of this FileShare less of a concern than it is for the shared-store use case?
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https://www.redpointglobal.com/&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
-----Original Message-----
From: Justin Bertram <jb...@apache.org>>
Sent: Friday, January 12, 2024 4:53 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
I sent that email on December 15. You can find it on the web-based mailing
list interface [1].
Justin
[1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l<https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l>
On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <pa...@hpe.com>>
wrote:
> Hi Justin,
>
> Forgive me but I don’t remember a journal persistence issue discussion. I
> did an email search too and did not see that discussion either. Would you
> please refresh me on the discussion? I may have inadvertently deleted that?
>
> I do see data being replicated to my production slave/standby instance on
> initial startup of the two broker instances. Failover works as expected.
> But when the primary instance starts backup I do not see the active slave
> replicate to the starting primary and I do not see the active slave restart
> and go to the backup server state.
>
> What is the communication sequence when the primary starts up after it
> quit with the active backup broker? In other words, how is the active
> backup broker notified to start replication to the newly started primary
> broker?
>
> I ran the replicated-failback example again with TRACE level logging
> turned on and I do see the attemptFailback=false message in the
> target/server1/log/artemis.log file at startup, so it must be changed later
> or am I not understanding the purpose of this flag?
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> discoveryInitialWaitTimeout=10000}]
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost, b=null],
> nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost)
>
> From my production slave broker log
>
>
> 024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> Thanks,
> Paul
> From: Justin Bertram <jb...@apache.org>>
> Date: Friday, January 12, 2024 at 3:38 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > Is it possible to use jgroups instead of UDP for the cluster connection
> in the example?
>
> Yes. Check out examples/features/clustered/clustered-jgroups for an example
> configuration. Keep in mind that UDP, JGroups, etc. is only used for
> initial discovery. Once brokers are discovered then normal TCP connections
> are created.
>
> Before I continue looking through your logs, etc. can you confirm that you
> resolved the journal persistence issue I described in my previous email? If
> that's not resolved there's no chance for failback to occur.
>
>
> Justin
>
>
>
> On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>>
> wrote:
>
> > Hi Justin,
> >
> > I tried the example and it works. I turned up logging to get a success
> > path through the code to maybe see where things get de-railed. Is it
> > possible to use jgroups instead of UDP for the cluster connection in the
> > example? That would give me a better success signature.
> >
> > My standby instance has this in the broker.xml which is the same as in
> the
> > example:
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > I am seeing this series of log messages that suggests that failback is
> not
> > enabled. Is the attemptFailback=false the default at this point in
> startup
> > and it is changed later or it there something gone wrong at this point?
> >
> > 2024-01-11 18:24:14,066 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > QuorumManager(server=null) about node up =
> > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> > <mailto:
<mailto:%0b>> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> >
> > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > ClientSessionFactoryImpl received backup update for live/backup pair =
> > TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > / null but it didn't belong to TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,067 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Cluster Connected
> >
> > 2024-01-11 18:24:14,068 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting backup manager
> >
> > 2024-01-11 18:24:14,069 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > backup config ClusterConnectionConfiguration{name='my-cluster',
> address='',
> > connectorName='artemis', clientFailureCheckPeriod=30000,
> > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > clusterNotificationAttempts=2}
> >
> > 2024-01-11 18:24:14,081 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > 2024-01-11 18:24:14,083 TRACE
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> >
> > java.lang.Exception:trace<ftp://java.lang.Exception/trace>
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClientSessionFactoryImpl.java:252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ServerLocatorImpl.java:628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1>) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ClusterController.java:313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1<https://linkprotect.cudasvc.com/url?a=https://org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4 rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<https://linkprotect.cudasvc.com/url?a=https://SharedNothingBackupActivation.java:199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1>(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1<https://linkprotect.cudasvc.com/url?a=https://ActiveMQServerImpl.java:4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying reconnection attempt 0/0
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18> > > <mailto:connectorFactory<mailto:connectorFactory>
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18>
> >
> > and currentConnectorConfig: TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Connector NettyConnector
> > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using native epoll
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Started EPOLL Netty Connector version 4.1.86.Final to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Remote destination:
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.25:61616<ftp://10.44.0.25/61616>
> >
> >
> > Thanks,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>>
> > Date: Wednesday, December 13, 2023 at 3:23 PM
> > To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > ActiveMQ Artemis ships with an example in the
> > examples/features/ha/replicated-failback directory that demonstrates how
> > this works. I ran this example and it worked fine. I also started up the
> > brokers manually and triggered failover and failback. Again, everything
> > worked fine. My only guess is that there's something configured wrong or
> > perhaps there's an environmental issue that's preventing failback from
> > working properly for you. If you could provide clear steps on how to
> > reproduce this issue I could investigate further.
> >
> >
> > Justin
> >
> > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am having an issue with failback to the master broker. I start the
> > > master broker then the slave broker. I see the replication from the
> > master
> > > to the slave happen. I stop the master/active broker and a failover
> > happens
> > > to the slave as expected. When the master is started a failback to it
> is
> > > expected but does not happen. The following is my ha-policy config.
> > Did I
> > > miss something in the ha-policy configuration? We are using version
> > 2.28.0
> > > and java17.
> > >
> > > Thanks,
> > > Paul
> > >
> > >
> > > Master broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <master>
> > >
> > > <check-for-live-server>true</check-for-live-server>
> > >
> > > </master>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > Slave broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > The following are the logs from the master and slave brokers
> > > HA “master” broker log after restart
> > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > Starting embedded web server
> > > 2023-12-08 20:42:57,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 15, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,627 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3
> > > 2023-12-08 20:42:57,657 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2> > > > [ID=47edf16f<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2>
> > [ID=47edf16f>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/10.44.0.18:34508<http://10.44.0.18:34508>]]
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > > activemq-branding plugin
> > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > artemis-plugin plugin
> > > 2023-12-08 20:42:57,766 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 18, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,767 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 20, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > Initialising
> > > hawtio services
> > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > Configuration
> > > will be discovered via system properties
> > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > Hawtio 2.15.0
> > > 2023-12-08 20:42:57,975 INFO
> > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > authentication filter, JAAS realm: "activemq" authorized role(s):
> "admin"
> > > role principal classes:
> > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > servlet is disabled
> > > 2023-12-08 20:42:57,989 INFO
> > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > > property: [key=policyLocation,
> > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > HTTP Server started at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> >
> > >
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > Artemis Jolokia REST API available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08> > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08> > >
> > >
> > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > Console
> > > available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08>
> >
> > >
> > > 20:42:58,455 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl]
> > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d> > > > [ID=f85ec51b<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d>
> > [ID=f85ec51b>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:49865<http://127.0.0.6:49865>]]
> > > 2023-12-08 20:42:59,524 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858> > > > [ID=7cef23a2<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858>
> > [ID=7cef23a2>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:59691<http://127.0.0.6:59691>]]
> > > 2023-12-08 20:42:59,535 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 22, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,536 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,549 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 24, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,550 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,583 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: OK
> > > 2023-12-08 20:42:59,584 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@5f46053b
> > > 2023-12-08 20:42:59,593 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > 2023-12-08 20:42:59,594 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 28, result: OK
> > > 2023-12-08 20:42:59,616 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:59,615 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 31, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 32, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 35, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 39, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:43:08,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256> > > > [ID=1d1d79ce<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256>
> > [ID=1d1d79ce>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:60787<http://127.0.0.6:60787>]]
> > > 2023-12-08 20:43:08,464 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7> > > > [ID=8f88c6d5<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7>
> > [ID=8f88c6d5>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:40065<http://127.0.0.6:40065>]]
> > >
> > >
> > > HA “slave” broker log.
> > >
> > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > synchronized with live server,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > >
> > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221031: backup announced
> > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221084: Requested 0 quorum votes
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221071: Failing over based on quorum vote results.
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:51,946 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:56,977 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > destroyed
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2023-12-08 20:42:57,445 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6
> > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > > Retrying
> > > 2023-12-08 20:42:57,629 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > > [ID=483170f6<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:55911<http://127.0.0.6:55911>]]
> > > 2023-12-08 20:42:57,630 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > > [ID=4514a2b3<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:60199<http://127.0.0.6:60199>]]
> > > 2023-12-08 20:42:57,648 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > > [ID=fcde4d78<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:53221<http://127.0.0.6:53221>]]
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 29, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 31, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,708 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 33, result: OK
> > > 2023-12-08 20:42:57,709 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@3645037f
> > > 2023-12-08 20:42:57,719 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 35, result: OK
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:57,753 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > 2023-12-08 20:42:57,760 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 38, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, > > > with transaction: null
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 41, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 43, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 44, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 46, result: OK
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 48, result: OK
> > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > >
> > >
> > >
> >
>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
RE: Trouble with Replication HA Master/Slave config performing failback
Posted by John Lilley <jo...@redpointglobal.com.INVALID>.
Hi Justin,
Can you elaborate on the lock file in the <node-manager-lock-directory>?
Does it need to be specified in the broker.xml for both the master and the slave?
Does it actually get used as a “lock” in the concurrency sense, to coordinate the live/backup election?
Thanks
john
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>
Sent: Monday, January 15, 2024 9:56 PM
To: users@activemq.apache.org
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
In the replication case you still need a PV in K8s, but I don't believe you need to put the whole journal on the PV (and suffer the performance penalty). You should just need to point the node-manager-lock-directory in broker.xml to the PV.
Justin
On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
Based on your reply in the provided link, are you saying that even for the replication HA case, you still need an external FileShare PV, in K8s, so that the nodeID can be persisted between restarts. In this use case, is the performance of this FileShare less of a concern than it is for the shared-store use case?
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,wIHynQgNzLIzcwjariMyRLGUiOg2HFp89mhuKhW7jjTNJ_0g2kIAKOE2kLQ6HVoBn6I30wClJKDGGsiWjUqjAQcLF68WbhJztNf4ue2yZg,,&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
-----Original Message-----
From: Justin Bertram <jb...@apache.org>>
Sent: Friday, January 12, 2024 4:53 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
I sent that email on December 15. You can find it on the web-based mailing
list interface [1].
Justin
[1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l
On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <pa...@hpe.com>>
wrote:
> Hi Justin,
>
> Forgive me but I don’t remember a journal persistence issue discussion. I
> did an email search too and did not see that discussion either. Would you
> please refresh me on the discussion? I may have inadvertently deleted that?
>
> I do see data being replicated to my production slave/standby instance on
> initial startup of the two broker instances. Failover works as expected.
> But when the primary instance starts backup I do not see the active slave
> replicate to the starting primary and I do not see the active slave restart
> and go to the backup server state.
>
> What is the communication sequence when the primary starts up after it
> quit with the active backup broker? In other words, how is the active
> backup broker notified to start replication to the newly started primary
> broker?
>
> I ran the replicated-failback example again with TRACE level logging
> turned on and I do see the attemptFailback=false message in the
> target/server1/log/artemis.log file at startup, so it must be changed later
> or am I not understanding the purpose of this flag?
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> discoveryInitialWaitTimeout=10000}]
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost, b=null],
> nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost)
>
> From my production slave broker log
>
>
> 024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> Thanks,
> Paul
> From: Justin Bertram <jb...@apache.org>>
> Date: Friday, January 12, 2024 at 3:38 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > Is it possible to use jgroups instead of UDP for the cluster connection
> in the example?
>
> Yes. Check out examples/features/clustered/clustered-jgroups for an example
> configuration. Keep in mind that UDP, JGroups, etc. is only used for
> initial discovery. Once brokers are discovered then normal TCP connections
> are created.
>
> Before I continue looking through your logs, etc. can you confirm that you
> resolved the journal persistence issue I described in my previous email? If
> that's not resolved there's no chance for failback to occur.
>
>
> Justin
>
>
>
> On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>>
> wrote:
>
> > Hi Justin,
> >
> > I tried the example and it works. I turned up logging to get a success
> > path through the code to maybe see where things get de-railed. Is it
> > possible to use jgroups instead of UDP for the cluster connection in the
> > example? That would give me a better success signature.
> >
> > My standby instance has this in the broker.xml which is the same as in
> the
> > example:
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > I am seeing this series of log messages that suggests that failback is
> not
> > enabled. Is the attemptFailback=false the default at this point in
> startup
> > and it is changed later or it there something gone wrong at this point?
> >
> > 2024-01-11 18:24:14,066 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > QuorumManager(server=null) about node up =
> > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> > <mailto:
<mailto:%0b>> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> >
> > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > ClientSessionFactoryImpl received backup update for live/backup pair =
> > TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > / null but it didn't belong to TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,067 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Cluster Connected
> >
> > 2024-01-11 18:24:14,068 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting backup manager
> >
> > 2024-01-11 18:24:14,069 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > backup config ClusterConnectionConfiguration{name='my-cluster',
> address='',
> > connectorName='artemis', clientFailureCheckPeriod=30000,
> > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > clusterNotificationAttempts=2}
> >
> > 2024-01-11 18:24:14,081 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > 2024-01-11 18:24:14,083 TRACE
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> >
> > java.lang.Exception:trace<ftp://java.lang.Exception/trace>
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4 rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying reconnection attempt 0/0
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18> > > <mailto:connectorFactory<mailto:connectorFactory>
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18>
> >
> > and currentConnectorConfig: TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Connector NettyConnector
> > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using native epoll
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Started EPOLL Netty Connector version 4.1.86.Final to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Remote destination:
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.25:61616<ftp://10.44.0.25/61616>
> >
> >
> > Thanks,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>>
> > Date: Wednesday, December 13, 2023 at 3:23 PM
> > To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > ActiveMQ Artemis ships with an example in the
> > examples/features/ha/replicated-failback directory that demonstrates how
> > this works. I ran this example and it worked fine. I also started up the
> > brokers manually and triggered failover and failback. Again, everything
> > worked fine. My only guess is that there's something configured wrong or
> > perhaps there's an environmental issue that's preventing failback from
> > working properly for you. If you could provide clear steps on how to
> > reproduce this issue I could investigate further.
> >
> >
> > Justin
> >
> > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am having an issue with failback to the master broker. I start the
> > > master broker then the slave broker. I see the replication from the
> > master
> > > to the slave happen. I stop the master/active broker and a failover
> > happens
> > > to the slave as expected. When the master is started a failback to it
> is
> > > expected but does not happen. The following is my ha-policy config.
> > Did I
> > > miss something in the ha-policy configuration? We are using version
> > 2.28.0
> > > and java17.
> > >
> > > Thanks,
> > > Paul
> > >
> > >
> > > Master broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <master>
> > >
> > > <check-for-live-server>true</check-for-live-server>
> > >
> > > </master>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > Slave broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > The following are the logs from the master and slave brokers
> > > HA “master” broker log after restart
> > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > Starting embedded web server
> > > 2023-12-08 20:42:57,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 15, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,627 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3
> > > 2023-12-08 20:42:57,657 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2> > > > [ID=47edf16f<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2>
> > [ID=47edf16f>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/10.44.0.18:34508<http://10.44.0.18:34508>]]
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > > activemq-branding plugin
> > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > artemis-plugin plugin
> > > 2023-12-08 20:42:57,766 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 18, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,767 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 20, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > Initialising
> > > hawtio services
> > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > Configuration
> > > will be discovered via system properties
> > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > Hawtio 2.15.0
> > > 2023-12-08 20:42:57,975 INFO
> > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > authentication filter, JAAS realm: "activemq" authorized role(s):
> "admin"
> > > role principal classes:
> > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > servlet is disabled
> > > 2023-12-08 20:42:57,989 INFO
> > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > > property: [key=policyLocation,
> > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > HTTP Server started at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > >
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > Artemis Jolokia REST API available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > >
> > >
> > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > Console
> > > available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> > >
> > > 20:42:58,455 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl]
> > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d> > > > [ID=f85ec51b<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d>
> > [ID=f85ec51b>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:49865<http://127.0.0.6:49865>]]
> > > 2023-12-08 20:42:59,524 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858> > > > [ID=7cef23a2<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858>
> > [ID=7cef23a2>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:59691<http://127.0.0.6:59691>]]
> > > 2023-12-08 20:42:59,535 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 22, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,536 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,549 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 24, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,550 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,583 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: OK
> > > 2023-12-08 20:42:59,584 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@5f46053b
> > > 2023-12-08 20:42:59,593 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > 2023-12-08 20:42:59,594 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 28, result: OK
> > > 2023-12-08 20:42:59,616 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:59,615 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 31, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 32, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 35, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 39, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:43:08,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256> > > > [ID=1d1d79ce<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256>
> > [ID=1d1d79ce>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:60787<http://127.0.0.6:60787>]]
> > > 2023-12-08 20:43:08,464 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7> > > > [ID=8f88c6d5<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7>
> > [ID=8f88c6d5>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:40065<http://127.0.0.6:40065>]]
> > >
> > >
> > > HA “slave” broker log.
> > >
> > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > synchronized with live server,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > >
> > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221031: backup announced
> > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221084: Requested 0 quorum votes
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221071: Failing over based on quorum vote results.
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:51,946 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:56,977 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > destroyed
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2023-12-08 20:42:57,445 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6
> > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > > Retrying
> > > 2023-12-08 20:42:57,629 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > > [ID=483170f6<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:55911<http://127.0.0.6:55911>]]
> > > 2023-12-08 20:42:57,630 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > > [ID=4514a2b3<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:60199<http://127.0.0.6:60199>]]
> > > 2023-12-08 20:42:57,648 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > > [ID=fcde4d78<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:53221<http://127.0.0.6:53221>]]
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 29, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 31, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,708 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 33, result: OK
> > > 2023-12-08 20:42:57,709 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@3645037f
> > > 2023-12-08 20:42:57,719 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 35, result: OK
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:57,753 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > 2023-12-08 20:42:57,760 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 38, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, > > > with transaction: null
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 41, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 43, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 44, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 46, result: OK
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 48, result: OK
> > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > >
> > >
> > >
> >
>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
RE: Trouble with Replication HA Master/Slave config performing failback
Posted by Lino Pereira <li...@redpointglobal.com.INVALID>.
Hey Justin,
So, we have deployed our replication HA with the PV-persisted node-manager-lock-directory for the master lock file, and have been testing it on our K8S cluster. Unfortunately, we’ve run into an issue where, upon the failover from live-to-backup, one of our services is consistently hitting this AMQ219014 timeout use case, see exception below.
What is the best/recommended way for our code to handle this use case?
Specifically, should we retry on AMQ219014 exceptions?
Are there other AMQ exceptions that our code should also retry on?
Are there any broker config settings that we can change to alleviate this issue?
Is there a way to reduce the AMQ219014 timeout value down from 30-seconds, and is that recommended?
Thanks,
Lino
2024-01-18T22:46:13.436 [http-nio-9910-exec-7] RpcExceptionMapper.toResponse:78 [] ERROR - error sending message: AMQ219014: Timed out after waiting 30000 ms for response when sending packet 71
2024-01-18T22:46:13.436 [http-nio-9910-exec-7] RpcExceptionMapper.toResponse:79 [] INFO - Error in RPC response RpcException: httpCode=500, errorMessage=error sending message: AMQ219014: Timed out after waiting 30000 ms for response when sending packet 71
errorFrames:
javax.jms.JMSException: AMQ219014: Timed out after waiting 30000 ms for response when sending packet 71
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:550)
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:452)
at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:588)
at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:305)
at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:277)
at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:147)
at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129)
at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:483)
at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:221)
at net.redpoint.ipc.jms.JmsProducerPool.send_(JmsProducerPool.java:359)
at net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:295)
at net.redpoint.ipc.jms.JmsRpcClientChannel.sendRequest(JmsRpcClientChannel.java:228)
at net.redpoint.ipc.jms.JmsRpcClientChannel.invokeRaw(JmsRpcClientChannel.java:202)
at net.redpoint.ipc.jms.JmsRpcClientChannel.call(JmsRpcClientChannel.java:101)
at net.redpoint.ipc.clients.RpcClientBase._sync(RpcClientBase.java:169)
at net.redpoint.ipc.clients.RpcClientBase._rpc(RpcClientBase.java:237)
at net.redpoint.rpdm.ipc.clients.MessageLogsClient.getLogDetailsCount(MessageLogsClient.java:217)
at net.redpoint.rpdm.ipc.web_service_gateway.MessageLogsHttpServer.lambda$getLogDetailsCount$8(MessageLogsHttpServer.java:371)
at net.redpoint.ipc.SecurityControl.doAsNoThrow(SecurityControl.java:272)
at net.redpoint.rpdm.ipc.web_service_gateway.MessageLogsHttpServer.getLogDetailsCount(MessageLogsHttpServer.java:370)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:240)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:697)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:357)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:340)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1744)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219014: Timed out after waiting 30000 ms for response when sending packet 71]
... 64 more
at net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:301)
at net.redpoint.ipc.jms.JmsRpcClientChannel.sendRequest(JmsRpcClientChannel.java:228)
at net.redpoint.ipc.jms.JmsRpcClientChannel.invokeRaw(JmsRpcClientChannel.java:202)
at net.redpoint.ipc.jms.JmsRpcClientChannel.call(JmsRpcClientChannel.java:101)
at net.redpoint.ipc.clients.RpcClientBase._sync(RpcClientBase.java:169)
at net.redpoint.ipc.clients.RpcClientBase._rpc(RpcClientBase.java:237)
at net.redpoint.rpdm.ipc.clients.MessageLogsClient.getLogDetailsCount(MessageLogsClient.java:217)
at net.redpoint.rpdm.ipc.web_service_gateway.MessageLogsHttpServer.lambda$getLogDetailsCount$8(MessageLogsHttpServer.java:371)
at net.redpoint.ipc.SecurityControl.doAsNoThrow(SecurityControl.java:272)
at net.redpoint.rpdm.ipc.web_service_gateway.MessageLogsHttpServer.getLogDetailsCount(MessageLogsHttpServer.java:370)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:240)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:697)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:357)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:340)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1744)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.jms.JMSException: AMQ219014: Timed out after waiting 30000 ms for response when sending packet 71
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:550)
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:452)
at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:588)
at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:305)
at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:277)
at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:147)
at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129)
at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:483)
at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:221)
at net.redpoint.ipc.jms.JmsProducerPool.send_(JmsProducerPool.java:359)
at net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:295)
... 53 more
Caused by: ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219014: Timed out after waiting 30000 ms for response when sending packet 71]
... 64 more
[rg] <https://www.redpointglobal.com/>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>
Sent: Monday, January 15, 2024 9:56 PM
To: users@activemq.apache.org
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
In the replication case you still need a PV in K8s, but I don't believe you need to put the whole journal on the PV (and suffer the performance penalty). You should just need to point the node-manager-lock-directory in broker.xml to the PV.
Justin
On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
Based on your reply in the provided link, are you saying that even for the replication HA case, you still need an external FileShare PV, in K8s, so that the nodeID can be persisted between restarts. In this use case, is the performance of this FileShare less of a concern than it is for the shared-store use case?
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,e-CkglZ9jkR9hd3c_OLvrgKpuX794B4Vn7HzczK5Tqwj_PK3fDvrq1pZG4YDDj8SEeM1WU6f-_W65-xNi4-GBFvLKE2yzy53oalJ8-M512YyaV8SMHs05N9M&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
-----Original Message-----
From: Justin Bertram <jb...@apache.org>>
Sent: Friday, January 12, 2024 4:53 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
I sent that email on December 15. You can find it on the web-based mailing
list interface [1].
Justin
[1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l
On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <pa...@hpe.com>>
wrote:
> Hi Justin,
>
> Forgive me but I don’t remember a journal persistence issue discussion. I
> did an email search too and did not see that discussion either. Would you
> please refresh me on the discussion? I may have inadvertently deleted that?
>
> I do see data being replicated to my production slave/standby instance on
> initial startup of the two broker instances. Failover works as expected.
> But when the primary instance starts backup I do not see the active slave
> replicate to the starting primary and I do not see the active slave restart
> and go to the backup server state.
>
> What is the communication sequence when the primary starts up after it
> quit with the active backup broker? In other words, how is the active
> backup broker notified to start replication to the newly started primary
> broker?
>
> I ran the replicated-failback example again with TRACE level logging
> turned on and I do see the attemptFailback=false message in the
> target/server1/log/artemis.log file at startup, so it must be changed later
> or am I not understanding the purpose of this flag?
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> discoveryInitialWaitTimeout=10000}]
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost, b=null],
> nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost)
>
> From my production slave broker log
>
>
> 024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> Thanks,
> Paul
> From: Justin Bertram <jb...@apache.org>>
> Date: Friday, January 12, 2024 at 3:38 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > Is it possible to use jgroups instead of UDP for the cluster connection
> in the example?
>
> Yes. Check out examples/features/clustered/clustered-jgroups for an example
> configuration. Keep in mind that UDP, JGroups, etc. is only used for
> initial discovery. Once brokers are discovered then normal TCP connections
> are created.
>
> Before I continue looking through your logs, etc. can you confirm that you
> resolved the journal persistence issue I described in my previous email? If
> that's not resolved there's no chance for failback to occur.
>
>
> Justin
>
>
>
> On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>>
> wrote:
>
> > Hi Justin,
> >
> > I tried the example and it works. I turned up logging to get a success
> > path through the code to maybe see where things get de-railed. Is it
> > possible to use jgroups instead of UDP for the cluster connection in the
> > example? That would give me a better success signature.
> >
> > My standby instance has this in the broker.xml which is the same as in
> the
> > example:
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > I am seeing this series of log messages that suggests that failback is
> not
> > enabled. Is the attemptFailback=false the default at this point in
> startup
> > and it is changed later or it there something gone wrong at this point?
> >
> > 2024-01-11 18:24:14,066 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > QuorumManager(server=null) about node up =
> > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> > <mailto:
<mailto:%0b>> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048>
> >
> > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > ClientSessionFactoryImpl received backup update for live/backup pair =
> > TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > / null but it didn't belong to TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,067 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Cluster Connected
> >
> > 2024-01-11 18:24:14,068 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting backup manager
> >
> > 2024-01-11 18:24:14,069 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > backup config ClusterConnectionConfiguration{name='my-cluster',
> address='',
> > connectorName='artemis', clientFailureCheckPeriod=30000,
> > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > clusterNotificationAttempts=2}
> >
> > 2024-01-11 18:24:14,081 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > 2024-01-11 18:24:14,083 TRACE
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> >
> > java.lang.Exception:trace<ftp://java.lang.Exception/trace>
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4 rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying reconnection attempt 0/0
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18> > > <mailto:connectorFactory<mailto:connectorFactory>
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18>
> >
> > and currentConnectorConfig: TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Connector NettyConnector
> > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using native epoll
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Started EPOLL Netty Connector version 4.1.86.Final to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Remote destination:
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.25:61616<ftp://10.44.0.25/61616>
> >
> >
> > Thanks,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>>
> > Date: Wednesday, December 13, 2023 at 3:23 PM
> > To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > ActiveMQ Artemis ships with an example in the
> > examples/features/ha/replicated-failback directory that demonstrates how
> > this works. I ran this example and it worked fine. I also started up the
> > brokers manually and triggered failover and failback. Again, everything
> > worked fine. My only guess is that there's something configured wrong or
> > perhaps there's an environmental issue that's preventing failback from
> > working properly for you. If you could provide clear steps on how to
> > reproduce this issue I could investigate further.
> >
> >
> > Justin
> >
> > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am having an issue with failback to the master broker. I start the
> > > master broker then the slave broker. I see the replication from the
> > master
> > > to the slave happen. I stop the master/active broker and a failover
> > happens
> > > to the slave as expected. When the master is started a failback to it
> is
> > > expected but does not happen. The following is my ha-policy config.
> > Did I
> > > miss something in the ha-policy configuration? We are using version
> > 2.28.0
> > > and java17.
> > >
> > > Thanks,
> > > Paul
> > >
> > >
> > > Master broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <master>
> > >
> > > <check-for-live-server>true</check-for-live-server>
> > >
> > > </master>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > Slave broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > The following are the logs from the master and slave brokers
> > > HA “master” broker log after restart
> > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > Starting embedded web server
> > > 2023-12-08 20:42:57,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 15, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,627 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3
> > > 2023-12-08 20:42:57,657 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2> > > > [ID=47edf16f<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2<ma...@556a5ff2>
> > [ID=47edf16f>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/10.44.0.18:34508<http://10.44.0.18:34508>]]
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > > activemq-branding plugin
> > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > artemis-plugin plugin
> > > 2023-12-08 20:42:57,766 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 18, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,767 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 20, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > Initialising
> > > hawtio services
> > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > Configuration
> > > will be discovered via system properties
> > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > Hawtio 2.15.0
> > > 2023-12-08 20:42:57,975 INFO
> > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > authentication filter, JAAS realm: "activemq" authorized role(s):
> "admin"
> > > role principal classes:
> > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > servlet is disabled
> > > 2023-12-08 20:42:57,989 INFO
> > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > > property: [key=policyLocation,
> > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > HTTP Server started at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > >
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > Artemis Jolokia REST API available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > >
> > >
> > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > Console
> > > available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> > >
> > > 20:42:58,455 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl]
> > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d> > > > [ID=f85ec51b<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d<ma...@1cb2285d>
> > [ID=f85ec51b>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:49865<http://127.0.0.6:49865>]]
> > > 2023-12-08 20:42:59,524 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858> > > > [ID=7cef23a2<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858<ma...@5f45858>
> > [ID=7cef23a2>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:59691<http://127.0.0.6:59691>]]
> > > 2023-12-08 20:42:59,535 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 22, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,536 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,549 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 24, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,550 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,583 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: OK
> > > 2023-12-08 20:42:59,584 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@5f46053b
> > > 2023-12-08 20:42:59,593 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > 2023-12-08 20:42:59,594 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 28, result: OK
> > > 2023-12-08 20:42:59,616 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:59,615 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 31, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 32, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 35, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 39, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:43:08,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256> > > > [ID=1d1d79ce<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256<ma...@51aa6256>
> > [ID=1d1d79ce>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:60787<http://127.0.0.6:60787>]]
> > > 2023-12-08 20:43:08,464 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7> > > > [ID=8f88c6d5<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7<ma...@14a6e5e7>
> > [ID=8f88c6d5>,
> > > local= /10.44.0.18:61616<http://10.44.0.18:61616>, remote=/127.0.0.6:40065<http://127.0.0.6:40065>]]
> > >
> > >
> > > HA “slave” broker log.
> > >
> > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > synchronized with live server,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > >
> > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221031: backup announced
> > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221084: Requested 0 quorum votes
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221071: Failing over based on quorum vote results.
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:51,946 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:56,977 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > destroyed
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:56167<http://127.0.0.6:56167>]]
> > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2023-12-08 20:42:57,445 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6
> > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > > Retrying
> > > 2023-12-08 20:42:57,629 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > > [ID=483170f6<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:55911<http://127.0.0.6:55911>]]
> > > 2023-12-08 20:42:57,630 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > > [ID=4514a2b3<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:60199<http://127.0.0.6:60199>]]
> > > 2023-12-08 20:42:57,648 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > > [ID=fcde4d78<mailto:transportConnection<mailto:transportConnection>
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78>,
> > > local= /10.40.0.106:61616<http://10.40.0.106:61616>, remote=/127.0.0.6:53221<http://127.0.0.6:53221>]]
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 29, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 31, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,708 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 33, result: OK
> > > 2023-12-08 20:42:57,709 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@3645037f
> > > 2023-12-08 20:42:57,719 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 35, result: OK
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:57,753 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > 2023-12-08 20:42:57,760 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 38, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, > > > with transaction: null
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 41, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 43, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 44, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 46, result: OK
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 48, result: OK
> > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > >
> > >
> > >
> >
>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by Justin Bertram <jb...@apache.org>.
In the replication case you still need a PV in K8s, but I don't believe you
need to put the whole journal on the PV (and suffer the performance
penalty). You should just need to point the node-manager-lock-directory in
broker.xml to the PV.
Justin
On Mon, Jan 15, 2024 at 4:35 PM Lino Pereira
<li...@redpointglobal.com.invalid> wrote:
> Hi Justin,
>
> Based on your reply in the provided link, are you saying that even for the
> replication HA case, you still need an external FileShare PV, in K8s, so
> that the nodeID can be persisted between restarts. In this use case, is the
> performance of this FileShare less of a concern than it is for the
> shared-store use case?
>
> Thanks,
> Lino
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> Lino Pereira
>
> C++ Developer, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Friday, January 12, 2024 4:53 PM
> To: users@activemq.apache.org
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
>
> *** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ***
>
> I sent that email on December 15. You can find it on the web-based mailing
> list interface [1].
>
> Justin
>
> [1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l
>
> On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <
> paul.shields@hpe.com>
> wrote:
>
> > Hi Justin,
> >
> > Forgive me but I don’t remember a journal persistence issue discussion. I
> > did an email search too and did not see that discussion either. Would you
> > please refresh me on the discussion? I may have inadvertently deleted
> that?
> >
> > I do see data being replicated to my production slave/standby instance on
> > initial startup of the two broker instances. Failover works as expected.
> > But when the primary instance starts backup I do not see the active slave
> > replicate to the starting primary and I do not see the active slave
> restart
> > and go to the backup server state.
> >
> > What is the communication sequence when the primary starts up after it
> > quit with the active backup broker? In other words, how is the active
> > backup broker notified to start replication to the newly started primary
> > broker?
> >
> > I ran the replicated-failback example again with TRACE level logging
> > turned on and I do see the attemptFailback=false message in the
> > target/server1/log/artemis.log file at startup, so it must be changed
> later
> > or am I not understanding the purpose of this flag?
> >
> > 2024-01-12 14:40:18,341 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> > discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=localhost, b=null],
> > nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
> >
> > 2024-01-12 14:40:18,341 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
> >
> > 2024-01-12 14:40:18,341 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=localhost)
> >
> > From my production slave broker log
> >
> >
> > 024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > Thanks,
> > Paul
> > From: Justin Bertram <jb...@apache.org>
> > Date: Friday, January 12, 2024 at 3:38 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > > Is it possible to use jgroups instead of UDP for the cluster connection
> > in the example?
> >
> > Yes. Check out examples/features/clustered/clustered-jgroups for an
> example
> > configuration. Keep in mind that UDP, JGroups, etc. is *only* used for
> > initial discovery. Once brokers are discovered then normal TCP
> connections
> > are created.
> >
> > Before I continue looking through your logs, etc. can you confirm that
> you
> > resolved the journal persistence issue I described in my previous email?
> If
> > that's not resolved there's no chance for failback to occur.
> >
> >
> > Justin
> >
> >
> >
> > On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> > paul.shields@hpe.com>
> > wrote:
> >
> > > Hi Justin,
> > >
> > > I tried the example and it works. I turned up logging to get a success
> > > path through the code to maybe see where things get de-railed. Is it
> > > possible to use jgroups instead of UDP for the cluster connection in
> the
> > > example? That would give me a better success signature.
> > >
> > > My standby instance has this in the broker.xml which is the same as in
> > the
> > > example:
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > I am seeing this series of log messages that suggests that failback is
> > not
> > > enabled. Is the attemptFailback=false the default at this point in
> > startup
> > > and it is changed later or it there something gone wrong at this point?
> > >
> > > 2024-01-11 18:24:14,066 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.Topology]
> Topology@7e0a913c
> > [owner=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > >
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > > QuorumManager(server=null) about node up =
> > > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > > Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null]
> > >
> > > 2024-01-11 18:24:14,067 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.Topology]
> Topology@7e0a913c
> > [owner=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > >
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > >
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> > > <mailto:
> > >
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> > >
> > > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > > Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null]
> > >
> > > 2024-01-11 18:24:14,067 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > ClientSessionFactoryImpl received backup update for live/backup pair =
> > > TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > > / null but it didn't belong to TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > >
> > > 2024-01-11 18:24:14,067 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Cluster Connected
> > >
> > > 2024-01-11 18:24:14,068 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Starting backup manager
> > >
> > > 2024-01-11 18:24:14,069 DEBUG
> > > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > > backup config ClusterConnectionConfiguration{name='my-cluster',
> > address='',
> > > connectorName='artemis', clientFailureCheckPeriod=30000,
> > > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > > maxRetryInterval=32000, initialConnectAttempts=20,
> reconnectAttempts=-1,
> > > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > > staticConnectors=[], discoveryGroupName='my-discovery-group',
> maxHops=1,
> > > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > > clusterNotificationAttempts=2}
> > >
> > > 2024-01-11 18:24:14,081 DEBUG
> > > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > > BackupManager connecting to DiscoveryBackupConnector
> > > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> > >
> > > 2024-01-11 18:24:14,082 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Starting Backup Server
> > >
> > > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Setting server state as started
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > looking up the node through nodeLocator.locateNode()
> > >
> > > 2024-01-11 18:24:14,083 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Connecting towards a possible live, connection
> > > information=Pair[a=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> > >
> > > 2024-01-11 18:24:14,083 DEBUG
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > >
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > > Calling
> > >
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> > >
> > > 2024-01-11 18:24:14,083 TRACE
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> > >
> > > java.lang.Exception:trace <ftp://java.lang.Exception/trace>
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0
> <ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0
> <ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0
> <ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1)
> > > ~[artemis-core-client-2.28.0.jar:2.28.0
> <ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1)
> > > ~[artemis-server-2.28.0.jar:2.28.0
> <ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1)
> > > ~[artemis-server-2.28.0.jar:2.28.0
> <ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4
> rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>)
> > > ~[artemis-server-2.28.0.jar:2.28.0
> <ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > at
> > >
> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1
> (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1)
> > > ~[artemis-server-2.28.0.jar:2.28.0
> <ftp://artemis-server-2.28.0.jar/2.28.0>]
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > Trying reconnection attempt 0/0
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > > Trying to connect with
> > >
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> > > <mailto:connectorFactory
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> > >
> > > and currentConnectorConfig: TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Connector NettyConnector
> > > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> > useServlet=false,
> > > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > > using native epoll
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Started EPOLL Netty Connector version 4.1.86.Final to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> <ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> > >
> > > 2024-01-11 18:24:14,084 DEBUG
> > > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > > Remote destination:
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.25:61616 <ftp://10.44.0.25/61616>
> > >
> > >
> > > Thanks,
> > > Paul
> > >
> > > From: Justin Bertram <jb...@apache.org>
> > > Date: Wednesday, December 13, 2023 at 3:23 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > > failback
> > > ActiveMQ Artemis ships with an example in the
> > > examples/features/ha/replicated-failback directory that demonstrates
> how
> > > this works. I ran this example and it worked fine. I also started up
> the
> > > brokers manually and triggered failover and failback. Again, everything
> > > worked fine. My only guess is that there's something configured wrong
> or
> > > perhaps there's an environmental issue that's preventing failback from
> > > working properly for you. If you could provide clear steps on how to
> > > reproduce this issue I could investigate further.
> > >
> > >
> > > Justin
> > >
> > > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> > paul.shields@hpe.com
> > > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I am having an issue with failback to the master broker. I start the
> > > > master broker then the slave broker. I see the replication from the
> > > master
> > > > to the slave happen. I stop the master/active broker and a failover
> > > happens
> > > > to the slave as expected. When the master is started a failback to it
> > is
> > > > expected but does not happen. The following is my ha-policy config.
> > > Did I
> > > > miss something in the ha-policy configuration? We are using version
> > > 2.28.0
> > > > and java17.
> > > >
> > > > Thanks,
> > > > Paul
> > > >
> > > >
> > > > Master broker.xml config
> > > >
> > > > <ha-policy>
> > > >
> > > > <replication>
> > > >
> > > > <master>
> > > >
> > > > <check-for-live-server>true</check-for-live-server>
> > > >
> > > > </master>
> > > >
> > > > </replication>
> > > >
> > > > </ha-policy>
> > > >
> > > > Slave broker.xml config
> > > >
> > > > <ha-policy>
> > > >
> > > > <replication>
> > > >
> > > > <slave>
> > > >
> > > > <allow-failback>true</allow-failback>
> > > >
> > > > </slave>
> > > >
> > > > </replication>
> > > >
> > > > </ha-policy>
> > > >
> > > > The following are the logs from the master and slave brokers
> > > > HA “master” broker log after restart
> > > > 2023-12-08 20:42:57,509 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883
> <ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> > for
> > > > protocols [MQTT]
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > >
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> <ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> > for
> > > > protocols [CORE]
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221007: Server is now live
> > > > 2023-12-08 20:42:57,534 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > > Starting embedded web server
> > > > 2023-12-08 20:42:57,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 15, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,627 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3
> > > > 2023-12-08 20:42:57,657 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > > > [ID=47edf16f<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > > [ID=47edf16f>,
> > > > local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
> > > > 2023-12-08 20:42:57,720 INFO
> > > > [org.apache.activemq.hawtio.branding.PluginContextListener]
> Initialized
> > > > activemq-branding plugin
> > > > 2023-12-08 20:42:57,737 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is connected
> > > > 2023-12-08 20:42:57,755 INFO
> > > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > > artemis-plugin plugin
> > > > 2023-12-08 20:42:57,766 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 18, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,767 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 20, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > > Initialising
> > > > hawtio services
> > > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > > Configuration
> > > > will be discovered via system properties
> > > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > > Hawtio 2.15.0
> > > > 2023-12-08 20:42:57,975 INFO
> > > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > > authentication filter, JAAS realm: "activemq" authorized role(s):
> > "admin"
> > > > role principal classes:
> > > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > > servlet is disabled
> > > > 2023-12-08 20:42:57,989 INFO
> > > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia
> overridden
> > > > property: [key=policyLocation,
> > > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > > HTTP Server started at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> > >
> > > >
> > > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > > Artemis Jolokia REST API available at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> > >
> > > >
> > > > 2023-12-08
> > > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> > >
> > > >
> > > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > > Console
> > > > available at
> > > >
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> > >
> > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> > >
> > > >
> > > > 2023-12-08
> > > > <
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> > >
> > > >
> > > > 20:42:58,455 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl]
> > > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > > > [ID=f85ec51b<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > > [ID=f85ec51b>,
> > > > local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
> > > > 2023-12-08 20:42:59,524 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > > > [ID=7cef23a2<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > > [ID=7cef23a2>,
> > > > local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
> > > > 2023-12-08 20:42:59,535 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 22, result: NO_BINDINGS
> > > > 2023-12-08 20:42:59,536 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> > connectionID:
> > > > 7cef23a2
> > > > 2023-12-08 20:42:59,549 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 24, result: NO_BINDINGS
> > > > 2023-12-08 20:42:59,550 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> > connectionID:
> > > > 7cef23a2
> > > > 2023-12-08 20:42:59,583 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 26, result: OK
> > > > 2023-12-08 20:42:59,584 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=true]@5f46053b
> > > > 2023-12-08 20:42:59,593 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> > created
> > > > consumer with ID: 0, with session name:
> > > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > > 2023-12-08 20:42:59,594 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 28, result: OK
> > > > 2023-12-08 20:42:59,616 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > > 2023-12-08 20:42:59,615 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[
> *AMQ*RESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > > 2023-12-08 20:42:59,621 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 31, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,621 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[
> *AMQ*RoutingName=ExpiryQueue,*AMQ*Distance=0,*AMQ*Address=ExpiryQueue,
> *AMQ*NotifType=BINDING_ADDED,*AMQ*Binding_ID=7,*AMQ*
> FilterString=NULL-value,*AMQ*NotifTimestamp=1702068179613,*AMQ*ClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
> > > > with transaction: null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 32, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[
> *AMQ*RoutingName=DLQ,*AMQ*Distance=0,*AMQ*Address=DLQ,*AMQ*
> NotifType=BINDING_ADDED,*AMQ*Binding_ID=3,*AMQ*FilterString=NULL-value,
> *AMQ*NotifTimestamp=1702068179614,*AMQ*ClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
> > > > with transaction: null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 35, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[
> *AMQ*RESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction:
> null
> > > > 2023-12-08 20:42:59,622 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 39, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7,
> consumerID: 0
> > > > 2023-12-08 20:43:08,447 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > > > [ID=1d1d79ce<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > > [ID=1d1d79ce>,
> > > > local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
> > > > 2023-12-08 20:43:08,464 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > > > [ID=8f88c6d5<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > > [ID=8f88c6d5>,
> > > > local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
> > > >
> > > >
> > > > HA “slave” broker log.
> > > >
> > > > 2023-12-08 20:37:58,374 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0
> [null]
> > > > started, waiting live to fail before it gets active
> > > >
> > > > 2023-12-08 20:37:59,020 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > > synchronized with live server,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > > >
> > > > 2023-12-08 20:38:00,032 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221031: backup announced
> > > > 2023-12-08 20:42:27,415 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221084: Requested 0 quorum votes
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > > 2023-12-08 20:42:27,416 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221071: Failing over based on quorum vote results.
> > > > 2023-12-08 20:42:27,431 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212037: Connection failure to
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > > 10.44.0.20:61616 <ftp://10.44.0.20/61616> has been detected:
> AMQ219015: The connection was
> > > > disconnected because of server shutdown [code=DISCONNECTED]
> > > > 2023-12-08 20:42:27,431 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212037: Connection failure to
> > > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > > 10.44.0.20:61616 <ftp://10.44.0.20/61616> has been detected:
> AMQ219015: The connection was
> > > > disconnected because of server shutdown [code=DISCONNECTED]
> > > > 2023-12-08 20:42:27,434 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > > 2023-12-08 20:42:27,574 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > > 2023-12-08 20:42:27,575 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > > 2023-12-08 20:42:27,578 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > > 2023-12-08 20:42:27,578 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > > 2023-12-08 20:42:27,611 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221007: Server is now live
> > > > 2023-12-08 20:42:27,699 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883
> <ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> > for
> > > > protocols [MQTT]
> > > > 2023-12-08 20:42:27,717 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221020: Started EPOLL Acceptor at
> > > >
> cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> <ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> > for
> > > > protocols [CORE]
> > > > 2023-12-08 20:42:51,946 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > > [ID=df357d0e<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > [ID=df357d0e>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > > > 2023-12-08 20:42:56,977 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > > destroyed
> > > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > > [ID=df357d0e<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > [ID=df357d0e>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > > > 2023-12-08 20:42:57,443 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212004: Failed to connect to server.
> > > > 2023-12-08 20:42:57,445 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 26, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,447 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6
> > > > 2023-12-08 20:42:57,512 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > > failure detected. Unblocking a blocking call that will never get a
> > > response
> > > > 2023-12-08 20:42:57,513 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is unable to connect to
> destination.
> > > > Retrying
> > > > 2023-12-08 20:42:57,629 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > > > [ID=483170f6<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > > [ID=483170f6>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
> > > > 2023-12-08 20:42:57,630 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > > > [ID=4514a2b3<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > > [ID=4514a2b3>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
> > > > 2023-12-08 20:42:57,648 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> > created
> > > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > >
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > > > [ID=fcde4d78<mailto:transportConnection
> > > >
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > > [ID=fcde4d78>,
> > > > local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
> > > > 2023-12-08 20:42:57,663 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 29, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,663 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> > connectionID:
> > > > fcde4d78
> > > > 2023-12-08 20:42:57,679 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 31, result: NO_BINDINGS
> > > > 2023-12-08 20:42:57,679 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> > created
> > > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> > connectionID:
> > > > fcde4d78
> > > > 2023-12-08 20:42:57,708 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 33, result: OK
> > > > 2023-12-08 20:42:57,709 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> > created
> > > > queue:
> > > >
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=true]@3645037f
> > > > 2023-12-08 20:42:57,719 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> > created
> > > > consumer with ID: 0, with session name:
> > > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > > 2023-12-08 20:42:57,720 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 35, result: OK
> > > > 2023-12-08 20:42:57,755 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > > 2023-12-08 20:42:57,753 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[
> *AMQ*RESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > > 2023-12-08 20:42:57,760 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 38, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,761 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[
> *AMQ*RoutingName=DLQ,*AMQ*Distance=0,*AMQ*Address=DLQ,*AMQ*
> NotifType=BINDING_ADDED,*AMQ*Binding_ID=3,*AMQ*FilterString=NULL-value,
> *AMQ*NotifTimestamp=1702068177753,*AMQ*ClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
> > > > with transaction: null
> > > > 2023-12-08 20:42:57,761 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 41, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[
> *AMQ*RoutingName=ExpiryQueue,*AMQ*Distance=0,*AMQ*Address=ExpiryQueue,
> *AMQ*NotifType=BINDING_ADDED,*AMQ*Binding_ID=7,*AMQ*
> FilterString=NULL-value,*AMQ*NotifTimestamp=1702068177753,*AMQ*ClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
> > > > with transaction: null
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 43, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > > acknowledged message:
> > > >
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > > timestamp=0,expiration=0, durable=false,
> > > >
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[
> *AMQ*RESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction:
> null
> > > > 2023-12-08 20:42:57,762 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > > delivered
> > > > message with message ID: 44, to consumer on address:
> > > > activemq.notifications, queue:
> > > >
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f,
> consumerID: 0
> > > > 2023-12-08 20:42:59,607 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > >
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > > >
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > >
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > > postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=amq-broker],
> > > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > > connector=TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > > [initialConnectors=[TransportConfiguration(name=artemis,
> > > >
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > > >
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > > discoveryGroupConfiguration=null]] is connected
> > > > 2023-12-08 20:42:59,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 46, result: OK
> > > > 2023-12-08 20:42:59,625 INFO
> > > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010:
> routed
> > > > message with ID: 48, result: OK
> > > > 2023-12-08 20:43:07,437 WARN
> [org.apache.activemq.artemis.core.server]
> > > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > > failure detected. Unblocking a blocking call that will never get a
> > > response
> > > >
> > > >
> > > >
> > >
> >
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
RE: Trouble with Replication HA Master/Slave config performing failback
Posted by Lino Pereira <li...@redpointglobal.com.INVALID>.
Hi Justin,
Based on your reply in the provided link, are you saying that even for the replication HA case, you still need an external FileShare PV, in K8s, so that the nodeID can be persisted between restarts. In this use case, is the performance of this FileShare less of a concern than it is for the shared-store use case?
Thanks,
Lino
[rg] <https://www.redpointglobal.com/>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
-----Original Message-----
From: Justin Bertram <jb...@apache.org>>
Sent: Friday, January 12, 2024 4:53 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
I sent that email on December 15. You can find it on the web-based mailing
list interface [1].
Justin
[1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l
On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <pa...@hpe.com>>
wrote:
> Hi Justin,
>
> Forgive me but I don’t remember a journal persistence issue discussion. I
> did an email search too and did not see that discussion either. Would you
> please refresh me on the discussion? I may have inadvertently deleted that?
>
> I do see data being replicated to my production slave/standby instance on
> initial startup of the two broker instances. Failover works as expected.
> But when the primary instance starts backup I do not see the active slave
> replicate to the starting primary and I do not see the active slave restart
> and go to the backup server state.
>
> What is the communication sequence when the primary starts up after it
> quit with the active backup broker? In other words, how is the active
> backup broker notified to start replication to the newly started primary
> broker?
>
> I ran the replicated-failback example again with TRACE level logging
> turned on and I do see the attemptFailback=false message in the
> target/server1/log/artemis.log file at startup, so it must be changed later
> or am I not understanding the purpose of this flag?
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> discoveryInitialWaitTimeout=10000}]
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost, b=null],
> nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost)
>
> From my production slave broker log
>
>
> 024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> Thanks,
> Paul
> From: Justin Bertram <jb...@apache.org>>
> Date: Friday, January 12, 2024 at 3:38 PM
> To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > Is it possible to use jgroups instead of UDP for the cluster connection
> in the example?
>
> Yes. Check out examples/features/clustered/clustered-jgroups for an example
> configuration. Keep in mind that UDP, JGroups, etc. is only used for
> initial discovery. Once brokers are discovered then normal TCP connections
> are created.
>
> Before I continue looking through your logs, etc. can you confirm that you
> resolved the journal persistence issue I described in my previous email? If
> that's not resolved there's no chance for failback to occur.
>
>
> Justin
>
>
>
> On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>>
> wrote:
>
> > Hi Justin,
> >
> > I tried the example and it works. I turned up logging to get a success
> > path through the code to maybe see where things get de-railed. Is it
> > possible to use jgroups instead of UDP for the cluster connection in the
> > example? That would give me a better success signature.
> >
> > My standby instance has this in the broker.xml which is the same as in
> the
> > example:
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > I am seeing this series of log messages that suggests that failback is
> not
> > enabled. Is the attemptFailback=false the default at this point in
> startup
> > and it is changed later or it there something gone wrong at this point?
> >
> > 2024-01-11 18:24:14,066 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > QuorumManager(server=null) about node up =
> > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> > <mailto:
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> >
> > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > ClientSessionFactoryImpl received backup update for live/backup pair =
> > TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > / null but it didn't belong to TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,067 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Cluster Connected
> >
> > 2024-01-11 18:24:14,068 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting backup manager
> >
> > 2024-01-11 18:24:14,069 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > backup config ClusterConnectionConfiguration{name='my-cluster',
> address='',
> > connectorName='artemis', clientFailureCheckPeriod=30000,
> > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > clusterNotificationAttempts=2}
> >
> > 2024-01-11 18:24:14,081 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > 2024-01-11 18:24:14,083 TRACE
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> >
> > java.lang.Exception:trace<ftp://java.lang.Exception/trace>
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a846&c=E,1,CCRYD9asOwUO9WfztF7iYeTstIMcTzC1PzToDMj0EbrmX2UCrDAtORKi3KkbIeNCyngS9b8yN3Espjh69EqwU7zCBnH8r4y0QR9JRY_jmA,,&typo=1) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a252&c=E,1,UWnIwEtYN-Os30noUm80TEjKQiOmsXl1jXnQy4rkVg_Mm8pEjR42vAnpDjuGLxaY-t6mzoLNPTHhGijPkTpytvNjie3IC5OpNppGu6mvWofV&typo=1) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a610&c=E,1,vriKlmcWPN4P5DVpudFT4BD6wYmRsznKQY-aJUx5dUDeuqbQ0B2VpgXUS0rVx_3I71s9Y8t9XsgeIR1ZeyEBM2jnHdASl70n9RRWcouCee6yazcDqAIo&typo=1) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fServerLocatorImpl.java%3a628&c=E,1,yZA0ipuWpmPIBY8F3Y6JTV4_GbJo7sPd9oUJykEzBTWOClB24oHhFNGyNJVyU4q2vKiCq8EI6pqavdcUQ4YeYkQJJ-sdYdrA-9ozNVsiWN0,&typo=1) > > ~[artemis-core-client-2.28.0.jar:2.28.0<ftp://artemis-core-client-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClusterController.java%3a313&c=E,1,hQdcc7oc1tjMYHgfz96Agje1LexFtj3XPS7thuEqXZ0nb-a9oo4CebgIK5y6J6mpQZgBEit92vAGgbN0AxU514WfP1gx_8TZqGr4ScoNNW-rkzGKcDQPSTY,&typo=1) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a348&c=E,1,i8Ok1EuiD_R938NmaeR8WDOMKGg9vgyUWMRtpW_pJ3eYEKZHcAwICLV2rEh1XaYhPfrlVpNgyzxMr9ycx65H-0UWqEi5Wg3fgsndcGZyuIdTsNgpyvvXD76mefY,&typo=1) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run&c=E,1,tQLZyyhJ9jYhtxt5fWofp5RfJOuoNSMT9BrpaC1Kfq-XAO8pGRZaU_gpz9u2opMkMpr-C2SmX2n_Y96jt8rsp5g4EdC8qXoj8i8mjSB2EyonliwDOUod-Q,,&typo=1(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4 rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fSharedNothingBackupActivation.java%3a199&c=E,1,3aR-ZwQ0L12WO6RcuG85ekB-kFU_DBCFf8XP3VkQwsiRzaKP0Dr7yyr1D_SIRByeDOfjSXZS-sk01qGGFSbY4rdD44bwCYRLdceWEwpKomO4blh1UmHfapKb&typo=1>) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActivationThread.run&c=E,1,E_Lo507fBrshPuy7zCANtA8BFPJ8Dqdfs6ai5o8MUPXHFZrxHGvfWmNiMihyA9Z34jHm93WN2awSavXmoqRLEsNKcLsV91wPT-p6mX28_Zg3s0Z0F3mZYkDdHQdK&typo=1(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQServerImpl.java%3a4473&c=E,1,K6a3M2nqN4-XbzaIFsZcqt65q4f8RCxlytHn-3pKsz5nMyS3M4B8RT0M4eOPJv9HotCE27zfFlucw9ZliuGmoQXbV5t7kRaRNEJjwCwwSF-HHrjzwlECJfssqEo,&typo=1) > > ~[artemis-server-2.28.0.jar:2.28.0<ftp://artemis-server-2.28.0.jar/2.28.0>]
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying reconnection attempt 0/0
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18 > > <mailto:connectorFactory
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> >
> > and currentConnectorConfig: TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Connector NettyConnector
> > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using native epoll
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Started EPOLL Netty Connector version 4.1.86.Final to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Remote destination:
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.25:61616<ftp://10.44.0.25/61616>
> >
> >
> > Thanks,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>>
> > Date: Wednesday, December 13, 2023 at 3:23 PM
> > To: users@activemq.apache.org<ma...@activemq.apache.org> <us...@activemq.apache.org>>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > ActiveMQ Artemis ships with an example in the
> > examples/features/ha/replicated-failback directory that demonstrates how
> > this works. I ran this example and it worked fine. I also started up the
> > brokers manually and triggered failover and failback. Again, everything
> > worked fine. My only guess is that there's something configured wrong or
> > perhaps there's an environmental issue that's preventing failback from
> > working properly for you. If you could provide clear steps on how to
> > reproduce this issue I could investigate further.
> >
> >
> > Justin
> >
> > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> paul.shields@hpe.com<ma...@hpe.com>
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am having an issue with failback to the master broker. I start the
> > > master broker then the slave broker. I see the replication from the
> > master
> > > to the slave happen. I stop the master/active broker and a failover
> > happens
> > > to the slave as expected. When the master is started a failback to it
> is
> > > expected but does not happen. The following is my ha-policy config.
> > Did I
> > > miss something in the ha-policy configuration? We are using version
> > 2.28.0
> > > and java17.
> > >
> > > Thanks,
> > > Paul
> > >
> > >
> > > Master broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <master>
> > >
> > > <check-for-live-server>true</check-for-live-server>
> > >
> > > </master>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > Slave broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > The following are the logs from the master and slave brokers
> > > HA “master” broker log after restart
> > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > Starting embedded web server
> > > 2023-12-08 20:42:57,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 15, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,627 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3
> > > 2023-12-08 20:42:57,657 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2 > > > [ID=47edf16f<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > [ID=47edf16f>,
> > > local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > > activemq-branding plugin
> > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > artemis-plugin plugin
> > > 2023-12-08 20:42:57,766 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 18, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,767 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 20, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > Initialising
> > > hawtio services
> > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > Configuration
> > > will be discovered via system properties
> > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > Hawtio 2.15.0
> > > 2023-12-08 20:42:57,975 INFO
> > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > authentication filter, JAAS realm: "activemq" authorized role(s):
> "admin"
> > > role principal classes:
> > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > servlet is disabled
> > > 2023-12-08 20:42:57,989 INFO
> > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > > property: [key=policyLocation,
> > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > HTTP Server started at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > >
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > Artemis Jolokia REST API available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 > >
> > >
> > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > Console
> > > available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> > >
> > > 20:42:58,455 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl]
> > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d > > > [ID=f85ec51b<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > [ID=f85ec51b>,
> > > local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
> > > 2023-12-08 20:42:59,524 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858 > > > [ID=7cef23a2<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > [ID=7cef23a2>,
> > > local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
> > > 2023-12-08 20:42:59,535 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 22, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,536 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,549 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 24, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,550 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,583 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: OK
> > > 2023-12-08 20:42:59,584 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@5f46053b
> > > 2023-12-08 20:42:59,593 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > 2023-12-08 20:42:59,594 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 28, result: OK
> > > 2023-12-08 20:42:59,616 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:59,615 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@967407085, > > > with transaction: null
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 31, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179613,AMQClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 32, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068179614,AMQClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 35, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@333704293, > > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 39, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:43:08,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256 > > > [ID=1d1d79ce<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > [ID=1d1d79ce>,
> > > local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
> > > 2023-12-08 20:43:08,464 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7 > > > [ID=8f88c6d5<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > [ID=8f88c6d5>,
> > > local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
> > >
> > >
> > > HA “slave” broker log.
> > >
> > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > synchronized with live server,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > >
> > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221031: backup announced
> > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221084: Requested 0 quorum votes
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221071: Failing over based on quorum vote results.
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616<ftp://10.44.0.20/61616> has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/1883>
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616<ftp://cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/61616>
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:51,946 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > > 2023-12-08 20:42:56,977 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > destroyed
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077 > > > [ID=df357d0e<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2023-12-08 20:42:57,445 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6
> > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > > Retrying
> > > 2023-12-08 20:42:57,629 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251 > > > [ID=483170f6<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6>,
> > > local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
> > > 2023-12-08 20:42:57,630 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b > > > [ID=4514a2b3<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3>,
> > > local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
> > > 2023-12-08 20:42:57,648 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8 > > > [ID=fcde4d78<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78>,
> > > local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 29, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 31, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,708 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 33, result: OK
> > > 2023-12-08 20:42:57,709 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@3645037f
> > > 2023-12-08 20:42:57,719 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 35, result: OK
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:57,753 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[AMQRESET_QUEUE_DATA=true]]@2125985563, > > > with transaction: null
> > > 2023-12-08 20:42:57,760 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 38, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[AMQRoutingName=DLQ,AMQDistance=0,AMQAddress=DLQ,AMQNotifType=BINDING_ADDED,AMQBinding_ID=3,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411, > > > with transaction: null
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 41, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[AMQRoutingName=ExpiryQueue,AMQDistance=0,AMQAddress=ExpiryQueue,AMQNotifType=BINDING_ADDED,AMQBinding_ID=7,AMQFilterString=NULL-value,AMQNotifTimestamp=1702068177753,AMQClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 43, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[AMQRESET_QUEUE_DATA_COMPLETE=true]]@1828250555, > > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 44, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 46, result: OK
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 48, result: OK
> > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > >
> > >
> > >
> >
>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by Justin Bertram <jb...@apache.org>.
I sent that email on December 15. You can find it on the web-based mailing
list interface [1].
Justin
[1] https://lists.apache.org/thread/5bv74br0rx5nxgk816tvbq53y51vch1l
On Fri, Jan 12, 2024 at 4:45 PM Shields, Paul Michael <pa...@hpe.com>
wrote:
> Hi Justin,
>
> Forgive me but I don’t remember a journal persistence issue discussion. I
> did an email search too and did not see that discussion either. Would you
> please refresh me on the discussion? I may have inadvertently deleted that?
>
> I do see data being replicated to my production slave/standby instance on
> initial startup of the two broker instances. Failover works as expected.
> But when the primary instance starts backup I do not see the active slave
> replicate to the starting primary and I do not see the active slave restart
> and go to the backup server state.
>
> What is the communication sequence when the primary starts up after it
> quit with the active backup broker? In other words, how is the active
> backup broker notified to start replication to the newly started primary
> broker?
>
> I ran the replicated-failback example again with TRACE level logging
> turned on and I do see the attemptFailback=false message in the
> target/server1/log/artemis.log file at startup, so it must be changed later
> or am I not understanding the purpose of this flag?
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000,
> discoveryInitialWaitTimeout=10000}]
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost, b=null],
> nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
>
> 2024-01-12 14:40:18,341 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=localhost)
>
> From my production slave broker log
>
>
> 024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> Thanks,
> Paul
> From: Justin Bertram <jb...@apache.org>
> Date: Friday, January 12, 2024 at 3:38 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> > Is it possible to use jgroups instead of UDP for the cluster connection
> in the example?
>
> Yes. Check out examples/features/clustered/clustered-jgroups for an example
> configuration. Keep in mind that UDP, JGroups, etc. is *only* used for
> initial discovery. Once brokers are discovered then normal TCP connections
> are created.
>
> Before I continue looking through your logs, etc. can you confirm that you
> resolved the journal persistence issue I described in my previous email? If
> that's not resolved there's no chance for failback to occur.
>
>
> Justin
>
>
>
> On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <
> paul.shields@hpe.com>
> wrote:
>
> > Hi Justin,
> >
> > I tried the example and it works. I turned up logging to get a success
> > path through the code to maybe see where things get de-railed. Is it
> > possible to use jgroups instead of UDP for the cluster connection in the
> > example? That would give me a better success signature.
> >
> > My standby instance has this in the broker.xml which is the same as in
> the
> > example:
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > I am seeing this series of log messages that suggests that failback is
> not
> > enabled. Is the attemptFailback=false the default at this point in
> startup
> > and it is changed later or it there something gone wrong at this point?
> >
> > 2024-01-11 18:24:14,066 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> > QuorumManager(server=null) about node up =
> > 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 TRACE
> > [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c
> [owner=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> >
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> > <mailto:
> >
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> >
> > about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> > Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null]
> >
> > 2024-01-11 18:24:14,067 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > ClientSessionFactoryImpl received backup update for live/backup pair =
> > TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> > / null but it didn't belong to TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,067 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Cluster Connected
> >
> > 2024-01-11 18:24:14,068 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting backup manager
> >
> > 2024-01-11 18:24:14,069 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> > backup config ClusterConnectionConfiguration{name='my-cluster',
> address='',
> > connectorName='artemis', clientFailureCheckPeriod=30000,
> > connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> > maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> > callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> > messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> > staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> > confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> > minLargeMessageSize=102400, clusterNotificationInterval=1000,
> > clusterNotificationAttempts=2}
> >
> > 2024-01-11 18:24:14,081 DEBUG
> > [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> > BackupManager connecting to DiscoveryBackupConnector
> > [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> > refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
> >
> > 2024-01-11 18:24:14,082 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Starting Backup Server
> >
> > 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Setting server state as started
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > looking up the node through nodeLocator.locateNode()
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Connecting towards a possible live, connection
> > information=Pair[a=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 DEBUG
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
> >
> > 2024-01-11 18:24:14,083 TRACE
> >
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> > Calling
> >
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
> >
> > 2024-01-11 18:24:14,083 TRACE
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
> >
> > java.lang.Exception: trace
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:846)
> > ~[artemis-core-client-2.28.0.jar:2.28.0]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:252)
> > ~[artemis-core-client-2.28.0.jar:2.28.0]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:610)
> > ~[artemis-core-client-2.28.0.jar:2.28.0]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:628)
> > ~[artemis-core-client-2.28.0.jar:2.28.0]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(ClusterController.java:313)
> > ~[artemis-server-2.28.0.jar:2.28.0]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(SharedNothingBackupActivation.java:348)
> > ~[artemis-server-2.28.0.jar:2.28.0]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:199)
> > ~[artemis-server-2.28.0.jar:2.28.0]
> >
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:4473)
> > ~[artemis-server-2.28.0.jar:2.28.0]
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying reconnection attempt 0/0
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> > Trying to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> > <mailto:connectorFactory
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> >
> > and currentConnectorConfig: TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Connector NettyConnector
> > [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> > port=61616, httpEnabled=false, httpUpgradeEnabled=false,
> useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using native epoll
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Started EPOLL Netty Connector version 4.1.86.Final to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> >
> > 2024-01-11 18:24:14,084 DEBUG
> > [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> > Remote destination:
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.25:61616
> >
> >
> > Thanks,
> > Paul
> >
> > From: Justin Bertram <jb...@apache.org>
> > Date: Wednesday, December 13, 2023 at 3:23 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Trouble with Replication HA Master/Slave config performing
> > failback
> > ActiveMQ Artemis ships with an example in the
> > examples/features/ha/replicated-failback directory that demonstrates how
> > this works. I ran this example and it worked fine. I also started up the
> > brokers manually and triggered failover and failback. Again, everything
> > worked fine. My only guess is that there's something configured wrong or
> > perhaps there's an environmental issue that's preventing failback from
> > working properly for you. If you could provide clear steps on how to
> > reproduce this issue I could investigate further.
> >
> >
> > Justin
> >
> > On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <
> paul.shields@hpe.com
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am having an issue with failback to the master broker. I start the
> > > master broker then the slave broker. I see the replication from the
> > master
> > > to the slave happen. I stop the master/active broker and a failover
> > happens
> > > to the slave as expected. When the master is started a failback to it
> is
> > > expected but does not happen. The following is my ha-policy config.
> > Did I
> > > miss something in the ha-policy configuration? We are using version
> > 2.28.0
> > > and java17.
> > >
> > > Thanks,
> > > Paul
> > >
> > >
> > > Master broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <master>
> > >
> > > <check-for-live-server>true</check-for-live-server>
> > >
> > > </master>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > Slave broker.xml config
> > >
> > > <ha-policy>
> > >
> > > <replication>
> > >
> > > <slave>
> > >
> > > <allow-failback>true</allow-failback>
> > >
> > > </slave>
> > >
> > > </replication>
> > >
> > > </ha-policy>
> > >
> > > The following are the logs from the master and slave brokers
> > > HA “master” broker log after restart
> > > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > > Starting embedded web server
> > > 2023-12-08 20:42:57,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 15, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,627 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3
> > > 2023-12-08 20:42:57,657 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > > [ID=47edf16f<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > [ID=47edf16f>,
> > > local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > > activemq-branding plugin
> > > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> > [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > > artemis-plugin plugin
> > > 2023-12-08 20:42:57,766 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 18, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,767 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 20, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> > Initialising
> > > hawtio services
> > > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> > Configuration
> > > will be discovered via system properties
> > > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > > Hawtio 2.15.0
> > > 2023-12-08 20:42:57,975 INFO
> > > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > > authentication filter, JAAS realm: "activemq" authorized role(s):
> "admin"
> > > role principal classes:
> > > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > > servlet is disabled
> > > 2023-12-08 20:42:57,989 INFO
> > > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > > property: [key=policyLocation,
> > > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > > HTTP Server started at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > >
> > > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > > Artemis Jolokia REST API available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> >
> > >
> > > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> > Console
> > > available at
> > >
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > >
> > > 2023-12-08
> > > <
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> > >
> > > 20:42:58,455 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl]
> > > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > > [ID=f85ec51b<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > [ID=f85ec51b>,
> > > local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
> > > 2023-12-08 20:42:59,524 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > > [ID=7cef23a2<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > [ID=7cef23a2>,
> > > local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
> > > 2023-12-08 20:42:59,535 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 22, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,536 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,549 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 24, result: NO_BINDINGS
> > > 2023-12-08 20:42:59,550 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session
> connectionID:
> > > 7cef23a2
> > > 2023-12-08 20:42:59,583 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: OK
> > > 2023-12-08 20:42:59,584 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@5f46053b
> > > 2023-12-08 20:42:59,593 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > > 2023-12-08 20:42:59,594 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 28, result: OK
> > > 2023-12-08 20:42:59,616 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:59,615 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@967407085,
> > > with transaction: null
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 31, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,621 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179613,_AMQ_ClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
> > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 32, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179614,_AMQ_ClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
> > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 35, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@333704293,
> > > with transaction: null
> > > 2023-12-08 20:42:59,622 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 39, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > > 2023-12-08 20:43:08,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > > [ID=1d1d79ce<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > [ID=1d1d79ce>,
> > > local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
> > > 2023-12-08 20:43:08,464 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > > [ID=8f88c6d5<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > [ID=8f88c6d5>,
> > > local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
> > >
> > >
> > > HA “slave” broker log.
> > >
> > > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > > started, waiting live to fail before it gets active
> > >
> > > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > > synchronized with live server,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> > >
> > > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221031: backup announced
> > > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221084: Requested 0 quorum votes
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221071: Failing over based on quorum vote results.
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212037: Connection failure to
> > > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > > 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> > > disconnected because of server shutdown [code=DISCONNECTED]
> > > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221007: Server is now live
> > > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883
> for
> > > protocols [MQTT]
> > > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221020: Started EPOLL Acceptor at
> > > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
> for
> > > protocols [CORE]
> > > 2023-12-08 20:42:51,946 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > [ID=df357d0e<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > > 2023-12-08 20:42:56,977 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> > destroyed
> > > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > > [ID=df357d0e<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e>,
> > > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2023-12-08 20:42:57,445 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 26, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,447 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6
> > > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > > Retrying
> > > 2023-12-08 20:42:57,629 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > > [ID=483170f6<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6>,
> > > local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
> > > 2023-12-08 20:42:57,630 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > > [ID=4514a2b3<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3>,
> > > local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
> > > 2023-12-08 20:42:57,648 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000:
> created
> > > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> > >
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > > [ID=fcde4d78<mailto:transportConnection
> > >
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78>,
> > > local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 29, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,663 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 31, result: NO_BINDINGS
> > > 2023-12-08 20:42:57,679 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002:
> created
> > > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session
> connectionID:
> > > fcde4d78
> > > 2023-12-08 20:42:57,708 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 33, result: OK
> > > 2023-12-08 20:42:57,709 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007:
> created
> > > queue:
> > >
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=true]@3645037f
> > > 2023-12-08 20:42:57,719 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005:
> created
> > > consumer with ID: 0, with session name:
> > 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > > 2023-12-08 20:42:57,720 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 35, result: OK
> > > 2023-12-08 20:42:57,755 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > > 2023-12-08 20:42:57,753 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@2125985563,
> > > with transaction: null
> > > 2023-12-08 20:42:57,760 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 38, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
> > > with transaction: null
> > > 2023-12-08 20:42:57,761 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 41, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
> > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 43, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > > acknowledged message:
> > >
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > > timestamp=0,expiration=0, durable=false,
> > >
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@1828250555,
> > > with transaction: null
> > > 2023-12-08 20:42:57,762 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> > delivered
> > > message with message ID: 44, to consumer on address:
> > > activemq.notifications, queue:
> > >
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> > >
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > >
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> > [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > > connector=TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > > [initialConnectors=[TransportConfiguration(name=artemis,
> > >
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > >
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > > discoveryGroupConfiguration=null]] is connected
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 46, result: OK
> > > 2023-12-08 20:42:59,625 INFO
> > > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > > message with ID: 48, result: OK
> > > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > > failure detected. Unblocking a blocking call that will never get a
> > response
> > >
> > >
> > >
> >
>
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by "Shields, Paul Michael" <pa...@hpe.com>.
Hi Justin,
Forgive me but I don’t remember a journal persistence issue discussion. I did an email search too and did not see that discussion either. Would you please refresh me on the discussion? I may have inadvertently deleted that?
I do see data being replicated to my production slave/standby instance on initial startup of the two broker instances. Failover works as expected. But when the primary instance starts backup I do not see the active slave replicate to the starting primary and I do not see the active slave restart and go to the backup server state.
What is the communication sequence when the primary starts up after it quit with the active backup broker? In other words, how is the active backup broker notified to start replication to the newly started primary broker?
I ran the replicated-failback example again with TRACE level logging turned on and I do see the attemptFailback=false message in the target/server1/log/artemis.log file at startup, so it must be changed later or am I not understanding the purpose of this flag?
2024-01-12 14:40:18,341 DEBUG [org.apache.activemq.artemis.core.server.cluster.BackupManager] ****** BackupManager connecting to DiscoveryBackupConnector [group=DiscoveryGroupConfiguration{name='dg-group1', refreshTimeout=5000, discoveryInitialWaitTimeout=10000}]
2024-01-12 14:40:18,341 DEBUG [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] Starting Backup Server
2024-01-12 14:40:18,341 INFO [org.apache.activemq.artemis.core.server] AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null] started, waiting live to fail before it gets active
2024-01-12 14:40:18,341 TRACE [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] Setting server state as started
2024-01-12 14:40:18,341 TRACE [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] looking up the node through nodeLocator.locateNode()
2024-01-12 14:40:18,341 DEBUG [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] Connecting towards a possible live, connection information=Pair[a=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=localhost, b=null], nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
2024-01-12 14:40:18,341 DEBUG [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] attemptFailback=false, nodeID=ab94924c-9f4e-11ee-95ee-569984593a04
2024-01-12 14:40:18,341 TRACE [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] Calling clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=localhost)
From my production slave broker log
024-01-11 18:24:14,082 DEBUG [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] Starting Backup Server
2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server] AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null] started, waiting live to fail before it gets active
2024-01-11 18:24:14,083 TRACE [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] Setting server state as started
2024-01-11 18:24:14,083 TRACE [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] looking up the node through nodeLocator.locateNode()
2024-01-11 18:24:14,083 DEBUG [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] Connecting towards a possible live, connection information=Pair[a=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local, b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
2024-01-11 18:24:14,083 DEBUG [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
2024-01-11 18:24:14,083 TRACE [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] Calling clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
Thanks,
Paul
From: Justin Bertram <jb...@apache.org>
Date: Friday, January 12, 2024 at 3:38 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
> Is it possible to use jgroups instead of UDP for the cluster connection
in the example?
Yes. Check out examples/features/clustered/clustered-jgroups for an example
configuration. Keep in mind that UDP, JGroups, etc. is *only* used for
initial discovery. Once brokers are discovered then normal TCP connections
are created.
Before I continue looking through your logs, etc. can you confirm that you
resolved the journal persistence issue I described in my previous email? If
that's not resolved there's no chance for failback to occur.
Justin
On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <pa...@hpe.com>
wrote:
> Hi Justin,
>
> I tried the example and it works. I turned up logging to get a success
> path through the code to maybe see where things get de-railed. Is it
> possible to use jgroups instead of UDP for the cluster connection in the
> example? That would give me a better success signature.
>
> My standby instance has this in the broker.xml which is the same as in the
> example:
>
> <ha-policy>
>
> <replication>
>
> <slave>
>
> <allow-failback>true</allow-failback>
>
> </slave>
>
> </replication>
>
> </ha-policy>
>
> I am seeing this series of log messages that suggests that failback is not
> enabled. Is the attemptFailback=false the default at this point in startup
> and it is changed later or it there something gone wrong at this point?
>
> 2024-01-11 18:24:14,066 TRACE
> [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c[owner=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> QuorumManager(server=null) about node up =
> 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null]
>
> 2024-01-11 18:24:14,067 TRACE
> [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c[owner=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> <mailto:
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048>
> about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null]
>
> 2024-01-11 18:24:14,067 DEBUG
> [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> ClientSessionFactoryImpl received backup update for live/backup pair =
> TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> / null but it didn't belong to TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
>
> 2024-01-11 18:24:14,067 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Cluster Connected
>
> 2024-01-11 18:24:14,068 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting backup manager
>
> 2024-01-11 18:24:14,069 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> backup config ClusterConnectionConfiguration{name='my-cluster', address='',
> connectorName='artemis', clientFailureCheckPeriod=30000,
> connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> minLargeMessageSize=102400, clusterNotificationInterval=1000,
> clusterNotificationAttempts=2}
>
> 2024-01-11 18:24:14,081 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
>
> 2024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
>
> java.lang.Exception: trace
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:846)
> ~[artemis-core-client-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:252)
> ~[artemis-core-client-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:610)
> ~[artemis-core-client-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:628)
> ~[artemis-core-client-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(ClusterController.java:313)
> ~[artemis-server-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(SharedNothingBackupActivation.java:348)
> ~[artemis-server-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:199)
> ~[artemis-server-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:4473)
> ~[artemis-server-2.28.0.jar:2.28.0]
>
> 2024-01-11 18:24:14,084 DEBUG
> [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> Trying reconnection attempt 0/0
>
> 2024-01-11 18:24:14,084 DEBUG
> [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> Trying to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> <mailto:connectorFactory
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18>
> and currentConnectorConfig: TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
>
> 2024-01-11 18:24:14,084 DEBUG
> [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> Connector NettyConnector
> [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using native epoll
>
> 2024-01-11 18:24:14,084 DEBUG
> [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> Started EPOLL Netty Connector version 4.1.86.Final to
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
>
> 2024-01-11 18:24:14,084 DEBUG
> [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> Remote destination:
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> 10.44.0.25:61616
>
>
> Thanks,
> Paul
>
> From: Justin Bertram <jb...@apache.org>
> Date: Wednesday, December 13, 2023 at 3:23 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> ActiveMQ Artemis ships with an example in the
> examples/features/ha/replicated-failback directory that demonstrates how
> this works. I ran this example and it worked fine. I also started up the
> brokers manually and triggered failover and failback. Again, everything
> worked fine. My only guess is that there's something configured wrong or
> perhaps there's an environmental issue that's preventing failback from
> working properly for you. If you could provide clear steps on how to
> reproduce this issue I could investigate further.
>
>
> Justin
>
> On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <paul.shields@hpe.com
> >
> wrote:
>
> > Hi,
> >
> > I am having an issue with failback to the master broker. I start the
> > master broker then the slave broker. I see the replication from the
> master
> > to the slave happen. I stop the master/active broker and a failover
> happens
> > to the slave as expected. When the master is started a failback to it is
> > expected but does not happen. The following is my ha-policy config.
> Did I
> > miss something in the ha-policy configuration? We are using version
> 2.28.0
> > and java17.
> >
> > Thanks,
> > Paul
> >
> >
> > Master broker.xml config
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <master>
> >
> > <check-for-live-server>true</check-for-live-server>
> >
> > </master>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > Slave broker.xml config
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > The following are the logs from the master and slave brokers
> > HA “master” broker log after restart
> > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> > protocols [MQTT]
> > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> > protocols [CORE]
> > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221007: Server is now live
> > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > Starting embedded web server
> > 2023-12-08 20:42:57,625 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 15, result: NO_BINDINGS
> > 2023-12-08 20:42:57,627 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> > queue:
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@15525ed3
> > 2023-12-08 20:42:57,657 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > [ID=47edf16f<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> [ID=47edf16f>,
> > local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
> > 2023-12-08 20:42:57,720 INFO
> > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > activemq-branding plugin
> > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > connector=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]] is connected
> > 2023-12-08 20:42:57,755 INFO
> > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > artemis-plugin plugin
> > 2023-12-08 20:42:57,766 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 18, result: NO_BINDINGS
> > 2023-12-08 20:42:57,767 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 20, result: NO_BINDINGS
> > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> Initialising
> > hawtio services
> > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> Configuration
> > will be discovered via system properties
> > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > Hawtio 2.15.0
> > 2023-12-08 20:42:57,975 INFO
> > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > authentication filter, JAAS realm: "activemq" authorized role(s): "admin"
> > role principal classes:
> > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > servlet is disabled
> > 2023-12-08 20:42:57,989 INFO
> > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > property: [key=policyLocation,
> > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > HTTP Server started at
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> >
> > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > Artemis Jolokia REST API available at
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> >
> > 2023-12-08
> > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08>
> >
> > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> Console
> > available at
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> >
> > 2023-12-08
> > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08>
> >
> > 20:42:58,455 INFO [org.apache.activemq.artemis.core.server.plugin.impl]
> > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > [ID=f85ec51b<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> [ID=f85ec51b>,
> > local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
> > 2023-12-08 20:42:59,524 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > [ID=7cef23a2<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> [ID=7cef23a2>,
> > local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
> > 2023-12-08 20:42:59,535 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 22, result: NO_BINDINGS
> > 2023-12-08 20:42:59,536 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session connectionID:
> > 7cef23a2
> > 2023-12-08 20:42:59,549 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 24, result: NO_BINDINGS
> > 2023-12-08 20:42:59,550 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session connectionID:
> > 7cef23a2
> > 2023-12-08 20:42:59,583 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 26, result: OK
> > 2023-12-08 20:42:59,584 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> > queue:
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=true]@5f46053b
> > 2023-12-08 20:42:59,593 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> > consumer with ID: 0, with session name:
> 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > 2023-12-08 20:42:59,594 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 28, result: OK
> > 2023-12-08 20:42:59,616 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > 2023-12-08 20:42:59,615 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > timestamp=0,expiration=0, durable=false,
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@967407085,
> > with transaction: null
> > 2023-12-08 20:42:59,621 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 31, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > 2023-12-08 20:42:59,621 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179613,_AMQ_ClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
> > with transaction: null
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 32, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179614,_AMQ_ClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
> > with transaction: null
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 35, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > timestamp=0,expiration=0, durable=false,
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@333704293,
> > with transaction: null
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 39, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > 2023-12-08 20:43:08,447 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > [ID=1d1d79ce<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> [ID=1d1d79ce>,
> > local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
> > 2023-12-08 20:43:08,464 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > [ID=8f88c6d5<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> [ID=8f88c6d5>,
> > local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
> >
> >
> > HA “slave” broker log.
> >
> > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > synchronized with live server,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> >
> > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221031: backup announced
> > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221084: Requested 0 quorum votes
> > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221071: Failing over based on quorum vote results.
> > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > AMQ212037: Connection failure to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> > disconnected because of server shutdown [code=DISCONNECTED]
> > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > AMQ212037: Connection failure to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> > disconnected because of server shutdown [code=DISCONNECTED]
> > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221007: Server is now live
> > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at
> > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> > protocols [MQTT]
> > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at
> > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> > protocols [CORE]
> > 2023-12-08 20:42:51,946 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > 2023-12-08 20:42:56,977 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> destroyed
> > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > AMQ212004: Failed to connect to server.
> > 2023-12-08 20:42:57,445 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 26, result: NO_BINDINGS
> > 2023-12-08 20:42:57,447 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> > queue:
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6
> > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > failure detected. Unblocking a blocking call that will never get a
> response
> > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > connector=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > Retrying
> > 2023-12-08 20:42:57,629 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> [ID=483170f6>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
> > 2023-12-08 20:42:57,630 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> [ID=4514a2b3>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
> > 2023-12-08 20:42:57,648 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> [ID=fcde4d78>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
> > 2023-12-08 20:42:57,663 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 29, result: NO_BINDINGS
> > 2023-12-08 20:42:57,663 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session connectionID:
> > fcde4d78
> > 2023-12-08 20:42:57,679 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 31, result: NO_BINDINGS
> > 2023-12-08 20:42:57,679 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session connectionID:
> > fcde4d78
> > 2023-12-08 20:42:57,708 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 33, result: OK
> > 2023-12-08 20:42:57,709 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> > queue:
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=true]@3645037f
> > 2023-12-08 20:42:57,719 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> > consumer with ID: 0, with session name:
> 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > 2023-12-08 20:42:57,720 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 35, result: OK
> > 2023-12-08 20:42:57,755 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > 2023-12-08 20:42:57,753 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > timestamp=0,expiration=0, durable=false,
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@2125985563,
> > with transaction: null
> > 2023-12-08 20:42:57,760 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 38, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > 2023-12-08 20:42:57,761 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
> > with transaction: null
> > 2023-12-08 20:42:57,761 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 41, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > 2023-12-08 20:42:57,762 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
> > with transaction: null
> > 2023-12-08 20:42:57,762 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 43, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > 2023-12-08 20:42:57,762 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > timestamp=0,expiration=0, durable=false,
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@1828250555,
> > with transaction: null
> > 2023-12-08 20:42:57,762 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 44, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > connector=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]] is connected
> > 2023-12-08 20:42:59,625 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 46, result: OK
> > 2023-12-08 20:42:59,625 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 48, result: OK
> > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > failure detected. Unblocking a blocking call that will never get a
> response
> >
> >
> >
>
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by Justin Bertram <jb...@apache.org>.
> Is it possible to use jgroups instead of UDP for the cluster connection
in the example?
Yes. Check out examples/features/clustered/clustered-jgroups for an example
configuration. Keep in mind that UDP, JGroups, etc. is *only* used for
initial discovery. Once brokers are discovered then normal TCP connections
are created.
Before I continue looking through your logs, etc. can you confirm that you
resolved the journal persistence issue I described in my previous email? If
that's not resolved there's no chance for failback to occur.
Justin
On Thu, Jan 11, 2024 at 3:09 PM Shields, Paul Michael <pa...@hpe.com>
wrote:
> Hi Justin,
>
> I tried the example and it works. I turned up logging to get a success
> path through the code to maybe see where things get de-railed. Is it
> possible to use jgroups instead of UDP for the cluster connection in the
> example? That would give me a better success signature.
>
> My standby instance has this in the broker.xml which is the same as in the
> example:
>
> <ha-policy>
>
> <replication>
>
> <slave>
>
> <allow-failback>true</allow-failback>
>
> </slave>
>
> </replication>
>
> </ha-policy>
>
> I am seeing this series of log messages that suggests that failback is not
> enabled. Is the attemptFailback=false the default at this point in startup
> and it is changed later or it there something gone wrong at this point?
>
> 2024-01-11 18:24:14,066 TRACE
> [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c[owner=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> QuorumManager(server=null) about node up =
> 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null]
>
> 2024-01-11 18:24:14,067 TRACE
> [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c[owner=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group',
> refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048
> <mailto:
> org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048>
> about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector =
> Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null]
>
> 2024-01-11 18:24:14,067 DEBUG
> [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> ClientSessionFactoryImpl received backup update for live/backup pair =
> TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
> / null but it didn't belong to TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
>
> 2024-01-11 18:24:14,067 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Cluster Connected
>
> 2024-01-11 18:24:14,068 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting backup manager
>
> 2024-01-11 18:24:14,069 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy
> backup config ClusterConnectionConfiguration{name='my-cluster', address='',
> connectorName='artemis', clientFailureCheckPeriod=30000,
> connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0,
> maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1,
> callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true,
> messageLoadBalancingType=ON_DEMAND, compositeMembers=null,
> staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1,
> confirmationWindowSize=10485760, allowDirectConnectionsOnly=false,
> minLargeMessageSize=102400, clusterNotificationInterval=1000,
> clusterNotificationAttempts=2}
>
> 2024-01-11 18:24:14,081 DEBUG
> [org.apache.activemq.artemis.core.server.cluster.BackupManager] ******
> BackupManager connecting to DiscoveryBackupConnector
> [group=DiscoveryGroupConfiguration{name='my-discovery-group',
> refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
>
> 2024-01-11 18:24:14,082 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Starting Backup Server
>
> 2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Setting server state as started
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> looking up the node through nodeLocator.locateNode()
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Connecting towards a possible live, connection
> information=Pair[a=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 DEBUG
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation]
> Calling
> clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
>
> 2024-01-11 18:24:14,083 TRACE
> [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
>
> java.lang.Exception: trace
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:846)
> ~[artemis-core-client-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:252)
> ~[artemis-core-client-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:610)
> ~[artemis-core-client-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:628)
> ~[artemis-core-client-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(ClusterController.java:313)
> ~[artemis-server-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(SharedNothingBackupActivation.java:348)
> ~[artemis-server-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:199)
> ~[artemis-server-2.28.0.jar:2.28.0]
>
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:4473)
> ~[artemis-server-2.28.0.jar:2.28.0]
>
> 2024-01-11 18:24:14,084 DEBUG
> [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> Trying reconnection attempt 0/0
>
> 2024-01-11 18:24:14,084 DEBUG
> [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl]
> Trying to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18
> <mailto:connectorFactory
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18>
> and currentConnectorConfig: TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
>
> 2024-01-11 18:24:14,084 DEBUG
> [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> Connector NettyConnector
> [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local,
> port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using native epoll
>
> 2024-01-11 18:24:14,084 DEBUG
> [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> Started EPOLL Netty Connector version 4.1.86.Final to
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
>
> 2024-01-11 18:24:14,084 DEBUG
> [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector]
> Remote destination:
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> 10.44.0.25:61616
>
>
> Thanks,
> Paul
>
> From: Justin Bertram <jb...@apache.org>
> Date: Wednesday, December 13, 2023 at 3:23 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Trouble with Replication HA Master/Slave config performing
> failback
> ActiveMQ Artemis ships with an example in the
> examples/features/ha/replicated-failback directory that demonstrates how
> this works. I ran this example and it worked fine. I also started up the
> brokers manually and triggered failover and failback. Again, everything
> worked fine. My only guess is that there's something configured wrong or
> perhaps there's an environmental issue that's preventing failback from
> working properly for you. If you could provide clear steps on how to
> reproduce this issue I could investigate further.
>
>
> Justin
>
> On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <paul.shields@hpe.com
> >
> wrote:
>
> > Hi,
> >
> > I am having an issue with failback to the master broker. I start the
> > master broker then the slave broker. I see the replication from the
> master
> > to the slave happen. I stop the master/active broker and a failover
> happens
> > to the slave as expected. When the master is started a failback to it is
> > expected but does not happen. The following is my ha-policy config.
> Did I
> > miss something in the ha-policy configuration? We are using version
> 2.28.0
> > and java17.
> >
> > Thanks,
> > Paul
> >
> >
> > Master broker.xml config
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <master>
> >
> > <check-for-live-server>true</check-for-live-server>
> >
> > </master>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > Slave broker.xml config
> >
> > <ha-policy>
> >
> > <replication>
> >
> > <slave>
> >
> > <allow-failback>true</allow-failback>
> >
> > </slave>
> >
> > </replication>
> >
> > </ha-policy>
> >
> > The following are the logs from the master and slave brokers
> > HA “master” broker log after restart
> > 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> > protocols [MQTT]
> > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> > protocols [CORE]
> > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221007: Server is now live
> > 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> > [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> > 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> > Starting embedded web server
> > 2023-12-08 20:42:57,625 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 15, result: NO_BINDINGS
> > 2023-12-08 20:42:57,627 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> > queue:
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@15525ed3
> > 2023-12-08 20:42:57,657 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> > [ID=47edf16f<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> [ID=47edf16f>,
> > local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
> > 2023-12-08 20:42:57,720 INFO
> > [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> > activemq-branding plugin
> > 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> >
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883
> [nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> > connector=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]] is connected
> > 2023-12-08 20:42:57,755 INFO
> > [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> > artemis-plugin plugin
> > 2023-12-08 20:42:57,766 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 18, result: NO_BINDINGS
> > 2023-12-08 20:42:57,767 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 20, result: NO_BINDINGS
> > 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener]
> Initialising
> > hawtio services
> > 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager]
> Configuration
> > will be discovered via system properties
> > 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> > Hawtio 2.15.0
> > 2023-12-08 20:42:57,975 INFO
> > [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> > authentication filter, JAAS realm: "activemq" authorized role(s): "admin"
> > role principal classes:
> > "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> > 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> > Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> > 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> > servlet is disabled
> > 2023-12-08 20:42:57,989 INFO
> > [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> > property: [key=policyLocation,
> > value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> > HTTP Server started at
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> >
> > 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> > Artemis Jolokia REST API available at
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> >
> > 2023-12-08
> > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08
> >
> > 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis
> Console
> > available at
> >
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> >
> > 2023-12-08
> > <
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08
> >
> > 20:42:58,455 INFO [org.apache.activemq.artemis.core.server.plugin.impl]
> > AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> > clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> > [ID=f85ec51b<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> [ID=f85ec51b>,
> > local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
> > 2023-12-08 20:42:59,524 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> > [ID=7cef23a2<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> [ID=7cef23a2>,
> > local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
> > 2023-12-08 20:42:59,535 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 22, result: NO_BINDINGS
> > 2023-12-08 20:42:59,536 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> > session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session connectionID:
> > 7cef23a2
> > 2023-12-08 20:42:59,549 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 24, result: NO_BINDINGS
> > 2023-12-08 20:42:59,550 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> > session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session connectionID:
> > 7cef23a2
> > 2023-12-08 20:42:59,583 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 26, result: OK
> > 2023-12-08 20:42:59,584 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> > queue:
> >
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=true]@5f46053b
> > 2023-12-08 20:42:59,593 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> > consumer with ID: 0, with session name:
> 5fc2919b-960a-11ee-8014-12a7ed0622c7
> > 2023-12-08 20:42:59,594 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 28, result: OK
> > 2023-12-08 20:42:59,616 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > message with ID: 29, result: OK, transaction: UNAVAILABLE
> > 2023-12-08 20:42:59,615 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> > timestamp=0,expiration=0, durable=false,
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@967407085,
> > with transaction: null
> > 2023-12-08 20:42:59,621 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 31, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > 2023-12-08 20:42:59,621 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179613,_AMQ_ClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
> > with transaction: null
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 32, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> > timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179614,_AMQ_ClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
> > with transaction: null
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 35, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> > timestamp=0,expiration=0, durable=false,
> >
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@333704293,
> > with transaction: null
> > 2023-12-08 20:42:59,622 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 39, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> > 2023-12-08 20:43:08,447 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> > [ID=1d1d79ce<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> [ID=1d1d79ce>,
> > local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
> > 2023-12-08 20:43:08,464 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> > nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> > [ID=8f88c6d5<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> [ID=8f88c6d5>,
> > local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
> >
> >
> > HA “slave” broker log.
> >
> > 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> > started, waiting live to fail before it gets active
> >
> > 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> > synchronized with live server,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
> >
> > 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221031: backup announced
> > 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221084: Requested 0 quorum votes
> > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221083: ignoring quorum vote as max cluster size is 1.
> > 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221071: Failing over based on quorum vote results.
> > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > AMQ212037: Connection failure to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> > disconnected because of server shutdown [code=DISCONNECTED]
> > 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> > AMQ212037: Connection failure to
> > cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> > 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> > disconnected because of server shutdown [code=DISCONNECTED]
> > 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> > 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> > 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> > 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> > 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221007: Server is now live
> > 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at
> > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> > protocols [MQTT]
> > 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at
> > cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> > protocols [CORE]
> > 2023-12-08 20:42:51,946 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > 2023-12-08 20:42:56,977 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001:
> destroyed
> > connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> > [ID=df357d0e<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> > 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> > AMQ212004: Failed to connect to server.
> > 2023-12-08 20:42:57,445 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 26, result: NO_BINDINGS
> > 2023-12-08 20:42:57,447 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> > queue:
> >
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6
> > 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > failure detected. Unblocking a blocking call that will never get a
> response
> > 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> > AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > connector=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]] is unable to connect to destination.
> > Retrying
> > 2023-12-08 20:42:57,629 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> > [ID=483170f6<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> [ID=483170f6>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
> > 2023-12-08 20:42:57,630 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> > [ID=4514a2b3<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> [ID=4514a2b3>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
> > 2023-12-08 20:42:57,648 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> > connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> > nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> >
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> > [ID=fcde4d78<mailto:transportConnection
> >
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> [ID=fcde4d78>,
> > local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
> > 2023-12-08 20:42:57,663 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 29, result: NO_BINDINGS
> > 2023-12-08 20:42:57,663 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> > session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session connectionID:
> > fcde4d78
> > 2023-12-08 20:42:57,679 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 31, result: NO_BINDINGS
> > 2023-12-08 20:42:57,679 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> > session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session connectionID:
> > fcde4d78
> > 2023-12-08 20:42:57,708 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 33, result: OK
> > 2023-12-08 20:42:57,709 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> > queue:
> >
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=true]@3645037f
> > 2023-12-08 20:42:57,719 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> > consumer with ID: 0, with session name:
> 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> > 2023-12-08 20:42:57,720 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 35, result: OK
> > 2023-12-08 20:42:57,755 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> > message with ID: 36, result: OK, transaction: UNAVAILABLE
> > 2023-12-08 20:42:57,753 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> > timestamp=0,expiration=0, durable=false,
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@2125985563,
> > with transaction: null
> > 2023-12-08 20:42:57,760 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 38, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > 2023-12-08 20:42:57,761 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
> > with transaction: null
> > 2023-12-08 20:42:57,761 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 41, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > 2023-12-08 20:42:57,762 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> > timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
> > with transaction: null
> > 2023-12-08 20:42:57,762 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 43, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > 2023-12-08 20:42:57,762 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> > acknowledged message:
> >
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> > timestamp=0,expiration=0, durable=false,
> >
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@1828250555,
> > with transaction: null
> > 2023-12-08 20:42:57,762 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012:
> delivered
> > message with message ID: 44, to consumer on address:
> > activemq.notifications, queue:
> >
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> > consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> > 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> >
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> >
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> > postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> > temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008
> [nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> > connector=TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> > address=, server=ActiveMQServerImpl::name=amq-broker]))
> > [initialConnectors=[TransportConfiguration(name=artemis,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> > discoveryGroupConfiguration=null]] is connected
> > 2023-12-08 20:42:59,625 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 46, result: OK
> > 2023-12-08 20:42:59,625 INFO
> > [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> > message with ID: 48, result: OK
> > 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> > AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> > failure detected. Unblocking a blocking call that will never get a
> response
> >
> >
> >
>
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by "Shields, Paul Michael" <pa...@hpe.com>.
Hi Justin,
I tried the example and it works. I turned up logging to get a success path through the code to maybe see where things get de-railed. Is it possible to use jgroups instead of UDP for the cluster connection in the example? That would give me a better success signature.
My standby instance has this in the broker.xml which is the same as in the example:
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
I am seeing this series of log messages that suggests that failback is not enabled. Is the attemptFailback=false the default at this point in startup and it is changed later or it there something gone wrong at this point?
2024-01-11 18:24:14,066 TRACE [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c[owner=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local], discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group', refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing QuorumManager(server=null) about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector = Pair[a=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local, b=null]
2024-01-11 18:24:14,067 TRACE [org.apache.activemq.artemis.core.client.impl.Topology] Topology@7e0a913c[owner=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local], discoveryGroupConfiguration=DiscoveryGroupConfiguration{name='my-discovery-group', refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]] informing org.apache.activemq.artemis.core.server.impl.AnyLiveNodeLocatorForReplication@43d39048<ma...@43d39048> about node up = 86d380d3-b0ae-11ee-9cff-7abbf21b3434 connector = Pair[a=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local, b=null]
2024-01-11 18:24:14,067 DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] ClientSessionFactoryImpl received backup update for live/backup pair = TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local / null but it didn't belong to TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
2024-01-11 18:24:14,067 TRACE [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] Cluster Connected
2024-01-11 18:24:14,068 DEBUG [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] Starting backup manager
2024-01-11 18:24:14,069 DEBUG [org.apache.activemq.artemis.core.server.cluster.BackupManager] deploy backup config ClusterConnectionConfiguration{name='my-cluster', address='', connectorName='artemis', clientFailureCheckPeriod=30000, connectionTTL=60000, retryInterval=1000, retryIntervalMultiplier=2.0, maxRetryInterval=32000, initialConnectAttempts=20, reconnectAttempts=-1, callTimeout=30000, callFailoverTimeout=-1, duplicateDetection=true, messageLoadBalancingType=ON_DEMAND, compositeMembers=null, staticConnectors=[], discoveryGroupName='my-discovery-group', maxHops=1, confirmationWindowSize=10485760, allowDirectConnectionsOnly=false, minLargeMessageSize=102400, clusterNotificationInterval=1000, clusterNotificationAttempts=2}
2024-01-11 18:24:14,081 DEBUG [org.apache.activemq.artemis.core.server.cluster.BackupManager] ****** BackupManager connecting to DiscoveryBackupConnector [group=DiscoveryGroupConfiguration{name='my-discovery-group', refreshTimeout=10000, discoveryInitialWaitTimeout=10000}]
2024-01-11 18:24:14,082 DEBUG [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] Starting Backup Server
2024-01-11 18:24:14,083 INFO [org.apache.activemq.artemis.core.server] AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null] started, waiting live to fail before it gets active
2024-01-11 18:24:14,083 TRACE [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] Setting server state as started
2024-01-11 18:24:14,083 TRACE [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] looking up the node through nodeLocator.locateNode()
2024-01-11 18:24:14,083 DEBUG [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] Connecting towards a possible live, connection information=Pair[a=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local, b=null], nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
2024-01-11 18:24:14,083 DEBUG [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] attemptFailback=false, nodeID=86d380d3-b0ae-11ee-9cff-7abbf21b3434
2024-01-11 18:24:14,083 TRACE [org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation] Calling clusterController.connectToNodeInReplicatedCluster(TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local)
2024-01-11 18:24:14,083 TRACE [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] getConnectionWithRetry::0 with retryInterval = 1000 multiplier = 2.0
java.lang.Exception: trace
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:846) ~[artemis-core-client-2.28.0.jar:2.28.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:252) ~[artemis-core-client-2.28.0.jar:2.28.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:610) ~[artemis-core-client-2.28.0.jar:2.28.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:628) ~[artemis-core-client-2.28.0.jar:2.28.0]
at org.apache.activemq.artemis.core.server.cluster.ClusterController.connectToNodeInReplicatedCluster(ClusterController.java:313) ~[artemis-server-2.28.0.jar:2.28.0]
at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.tryConnectToNodeInReplicatedCluster(SharedNothingBackupActivation.java:348) ~[artemis-server-2.28.0.jar:2.28.0]
at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:199) ~[artemis-server-2.28.0.jar:2.28.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:4473) ~[artemis-server-2.28.0.jar:2.28.0]
2024-01-11 18:24:14,084 DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] Trying reconnection attempt 0/0
2024-01-11 18:24:14,084 DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl] Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@66996f18<ma...@66996f18> and currentConnectorConfig: TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local
2024-01-11 18:24:14,084 DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector] Connector NettyConnector [host=cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using native epoll
2024-01-11 18:24:14,084 DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector] Started EPOLL Netty Connector version 4.1.86.Final to cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616
2024-01-11 18:24:14,084 DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector] Remote destination: cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/10.44.0.25:61616
Thanks,
Paul
From: Justin Bertram <jb...@apache.org>
Date: Wednesday, December 13, 2023 at 3:23 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Trouble with Replication HA Master/Slave config performing failback
ActiveMQ Artemis ships with an example in the
examples/features/ha/replicated-failback directory that demonstrates how
this works. I ran this example and it worked fine. I also started up the
brokers manually and triggered failover and failback. Again, everything
worked fine. My only guess is that there's something configured wrong or
perhaps there's an environmental issue that's preventing failback from
working properly for you. If you could provide clear steps on how to
reproduce this issue I could investigate further.
Justin
On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <pa...@hpe.com>
wrote:
> Hi,
>
> I am having an issue with failback to the master broker. I start the
> master broker then the slave broker. I see the replication from the master
> to the slave happen. I stop the master/active broker and a failover happens
> to the slave as expected. When the master is started a failback to it is
> expected but does not happen. The following is my ha-policy config. Did I
> miss something in the ha-policy configuration? We are using version 2.28.0
> and java17.
>
> Thanks,
> Paul
>
>
> Master broker.xml config
>
> <ha-policy>
>
> <replication>
>
> <master>
>
> <check-for-live-server>true</check-for-live-server>
>
> </master>
>
> </replication>
>
> </ha-policy>
>
> Slave broker.xml config
>
> <ha-policy>
>
> <replication>
>
> <slave>
>
> <allow-failback>true</allow-failback>
>
> </slave>
>
> </replication>
>
> </ha-policy>
>
> The following are the logs from the master and slave brokers
> HA “master” broker log after restart
> 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> protocols [MQTT]
> 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> protocols [CORE]
> 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> AMQ221007: Server is now live
> 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> Starting embedded web server
> 2023-12-08 20:42:57,625 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 15, result: NO_BINDINGS
> 2023-12-08 20:42:57,627 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@15525ed3
> 2023-12-08 20:42:57,657 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> [ID=47edf16f<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2[ID=47edf16f>,
> local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
> 2023-12-08 20:42:57,720 INFO
> [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> activemq-branding plugin
> 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883[nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> connector=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> address=, server=ActiveMQServerImpl::name=amq-broker]))
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]] is connected
> 2023-12-08 20:42:57,755 INFO
> [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> artemis-plugin plugin
> 2023-12-08 20:42:57,766 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 18, result: NO_BINDINGS
> 2023-12-08 20:42:57,767 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 20, result: NO_BINDINGS
> 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener] Initialising
> hawtio services
> 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager] Configuration
> will be discovered via system properties
> 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> Hawtio 2.15.0
> 2023-12-08 20:42:57,975 INFO
> [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> authentication filter, JAAS realm: "activemq" authorized role(s): "admin"
> role principal classes:
> "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> servlet is disabled
> 2023-12-08 20:42:57,989 INFO
> [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> property: [key=policyLocation,
> value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> HTTP Server started at
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161>
> 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> Artemis Jolokia REST API available at
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia>
> 2023-12-08
> <http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08 >
> 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console
> available at
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console<http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console>
> 2023-12-08
> <http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08 >
> 20:42:58,455 INFO [org.apache.activemq.artemis.core.server.plugin.impl]
> AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> [ID=f85ec51b<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d[ID=f85ec51b>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
> 2023-12-08 20:42:59,524 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> [ID=7cef23a2<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858[ID=7cef23a2>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
> 2023-12-08 20:42:59,535 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 22, result: NO_BINDINGS
> 2023-12-08 20:42:59,536 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session connectionID:
> 7cef23a2
> 2023-12-08 20:42:59,549 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 24, result: NO_BINDINGS
> 2023-12-08 20:42:59,550 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session connectionID:
> 7cef23a2
> 2023-12-08 20:42:59,583 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 26, result: OK
> 2023-12-08 20:42:59,584 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=true]@5f46053b
> 2023-12-08 20:42:59,593 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> consumer with ID: 0, with session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7
> 2023-12-08 20:42:59,594 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 28, result: OK
> 2023-12-08 20:42:59,616 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> message with ID: 29, result: OK, transaction: UNAVAILABLE
> 2023-12-08 20:42:59,615 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@967407085,
> with transaction: null
> 2023-12-08 20:42:59,621 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 31, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:42:59,621 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179613,_AMQ_ClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
> with transaction: null
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 32, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179614,_AMQ_ClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
> with transaction: null
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 35, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@333704293,
> with transaction: null
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 39, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:43:08,447 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> [ID=1d1d79ce<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256[ID=1d1d79ce>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
> 2023-12-08 20:43:08,464 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> [ID=8f88c6d5<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7[ID=8f88c6d5>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
>
>
> HA “slave” broker log.
>
> 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> synchronized with live server, nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
>
> 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> AMQ221031: backup announced
> 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> AMQ221084: Requested 0 quorum votes
> 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> AMQ221083: ignoring quorum vote as max cluster size is 1.
> 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> AMQ221071: Failing over based on quorum vote results.
> 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> AMQ212037: Connection failure to
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> disconnected because of server shutdown [code=DISCONNECTED]
> 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> AMQ212037: Connection failure to
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> disconnected because of server shutdown [code=DISCONNECTED]
> 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address DLQ supporting [ANYCAST]
> 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> AMQ221007: Server is now live
> 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> protocols [MQTT]
> 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> protocols [CORE]
> 2023-12-08 20:42:51,946 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> 2023-12-08 20:42:56,977 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001: destroyed
> connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> AMQ212004: Failed to connect to server.
> 2023-12-08 20:42:57,445 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 26, result: NO_BINDINGS
> 2023-12-08 20:42:57,447 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6
> 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> failure detected. Unblocking a blocking call that will never get a response
> 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008[nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> connector=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> address=, server=ActiveMQServerImpl::name=amq-broker]))
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]] is unable to connect to destination.
> Retrying
> 2023-12-08 20:42:57,629 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> [ID=483170f6<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251[ID=483170f6>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
> 2023-12-08 20:42:57,630 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> [ID=4514a2b3<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b[ID=4514a2b3>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
> 2023-12-08 20:42:57,648 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> [ID=fcde4d78<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8[ID=fcde4d78>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
> 2023-12-08 20:42:57,663 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 29, result: NO_BINDINGS
> 2023-12-08 20:42:57,663 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session connectionID:
> fcde4d78
> 2023-12-08 20:42:57,679 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 31, result: NO_BINDINGS
> 2023-12-08 20:42:57,679 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session connectionID:
> fcde4d78
> 2023-12-08 20:42:57,708 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 33, result: OK
> 2023-12-08 20:42:57,709 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=true]@3645037f
> 2023-12-08 20:42:57,719 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> consumer with ID: 0, with session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> 2023-12-08 20:42:57,720 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 35, result: OK
> 2023-12-08 20:42:57,755 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> message with ID: 36, result: OK, transaction: UNAVAILABLE
> 2023-12-08 20:42:57,753 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@2125985563,
> with transaction: null
> 2023-12-08 20:42:57,760 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 38, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:57,761 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
> with transaction: null
> 2023-12-08 20:42:57,761 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 41, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
> with transaction: null
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 43, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@1828250555,
> with transaction: null
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 44, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008[nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> connector=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> address=, server=ActiveMQServerImpl::name=amq-broker]))
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]] is connected
> 2023-12-08 20:42:59,625 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 46, result: OK
> 2023-12-08 20:42:59,625 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 48, result: OK
> 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> failure detected. Unblocking a blocking call that will never get a response
>
>
>
Re: Trouble with Replication HA Master/Slave config performing failback
Posted by Justin Bertram <jb...@apache.org>.
ActiveMQ Artemis ships with an example in the
examples/features/ha/replicated-failback directory that demonstrates how
this works. I ran this example and it worked fine. I also started up the
brokers manually and triggered failover and failback. Again, everything
worked fine. My only guess is that there's something configured wrong or
perhaps there's an environmental issue that's preventing failback from
working properly for you. If you could provide clear steps on how to
reproduce this issue I could investigate further.
Justin
On Fri, Dec 8, 2023 at 3:45 PM Shields, Paul Michael <pa...@hpe.com>
wrote:
> Hi,
>
> I am having an issue with failback to the master broker. I start the
> master broker then the slave broker. I see the replication from the master
> to the slave happen. I stop the master/active broker and a failover happens
> to the slave as expected. When the master is started a failback to it is
> expected but does not happen. The following is my ha-policy config. Did I
> miss something in the ha-policy configuration? We are using version 2.28.0
> and java17.
>
> Thanks,
> Paul
>
>
> Master broker.xml config
>
> <ha-policy>
>
> <replication>
>
> <master>
>
> <check-for-live-server>true</check-for-live-server>
>
> </master>
>
> </replication>
>
> </ha-policy>
>
> Slave broker.xml config
>
> <ha-policy>
>
> <replication>
>
> <slave>
>
> <allow-failback>true</allow-failback>
>
> </slave>
>
> </replication>
>
> </ha-policy>
>
> The following are the logs from the master and slave brokers
> HA “master” broker log after restart
> 2023-12-08 20:42:57,509 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> protocols [MQTT]
> 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> protocols [CORE]
> 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> AMQ221007: Server is now live
> 2023-12-08 20:42:57,534 INFO [org.apache.activemq.artemis.core.server]
> AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.28.0
> [amq-broker, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f]
> 2023-12-08 20:42:57,539 INFO [org.apache.activemq.artemis] AMQ241003:
> Starting embedded web server
> 2023-12-08 20:42:57,625 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 15, result: NO_BINDINGS
> 2023-12-08 20:42:57,627 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@15525ed3
> 2023-12-08 20:42:57,657 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=47edf16f, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2
> [ID=47edf16f<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@556a5ff2[ID=47edf16f>,
> local= /10.44.0.18:61616, remote=/10.44.0.18:34508]]
> 2023-12-08 20:42:57,720 INFO
> [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized
> activemq-branding plugin
> 2023-12-08 20:42:57,737 INFO [org.apache.activemq.artemis.core.server]
> AMQ221027: Bridge ClusterConnectionBridge@1bb2e6ff
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@1bb2e6ff
> [name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.9759f092-9609-11ee-9862-e26c98ea5364,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@15525ed3 targetConnector=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1409513883[nodeUUID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> connector=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> address=, server=ActiveMQServerImpl::name=amq-broker]))
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]] is connected
> 2023-12-08 20:42:57,755 INFO
> [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized
> artemis-plugin plugin
> 2023-12-08 20:42:57,766 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 18, result: NO_BINDINGS
> 2023-12-08 20:42:57,767 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 20, result: NO_BINDINGS
> 2023-12-08 20:42:57,968 INFO [io.hawt.HawtioContextListener] Initialising
> hawtio services
> 2023-12-08 20:42:57,970 INFO [io.hawt.system.ConfigManager] Configuration
> will be discovered via system properties
> 2023-12-08 20:42:57,971 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to
> Hawtio 2.15.0
> 2023-12-08 20:42:57,975 INFO
> [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio
> authentication filter, JAAS realm: "activemq" authorized role(s): "admin"
> role principal classes:
> "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
> 2023-12-08 20:42:57,979 INFO [io.hawt.web.auth.LoginRedirectFilter]
> Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
> 2023-12-08 20:42:57,986 INFO [io.hawt.web.proxy.ProxyServlet] Proxy
> servlet is disabled
> 2023-12-08 20:42:57,989 INFO
> [io.hawt.web.servlets.JolokiaConfiguredAgentServlet] Jolokia overridden
> property: [key=policyLocation,
> value=file:/home/jboss/amq-broker/etc/jolokia-access.xml]
> 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241001:
> HTTP Server started at
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161
> 2023-12-08 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241002:
> Artemis Jolokia REST API available at
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia
> 2023-12-08
> <http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console/jolokia2023-12-08>
> 20:42:58,058 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console
> available at
> http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console
> 2023-12-08
> <http://cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:8161/console2023-12-08>
> 20:42:58,455 INFO [org.apache.activemq.artemis.core.server.plugin.impl]
> AMQ841000: created connection: RemotingConnectionImpl [ID=f85ec51b,
> clientID=null, nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d
> [ID=f85ec51b<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@1cb2285d[ID=f85ec51b>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:49865]]
> 2023-12-08 20:42:59,524 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=7cef23a2, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858
> [ID=7cef23a2<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@5f45858[ID=7cef23a2>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:59691]]
> 2023-12-08 20:42:59,535 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 22, result: NO_BINDINGS
> 2023-12-08 20:42:59,536 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5fbf845a-960a-11ee-8014-12a7ed0622c7, session connectionID:
> 7cef23a2
> 2023-12-08 20:42:59,549 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 24, result: NO_BINDINGS
> 2023-12-08 20:42:59,550 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7, session connectionID:
> 7cef23a2
> 2023-12-08 20:42:59,583 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 26, result: OK
> 2023-12-08 20:42:59,584 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=true]@5f46053b
> 2023-12-08 20:42:59,593 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> consumer with ID: 0, with session name: 5fc2919b-960a-11ee-8014-12a7ed0622c7
> 2023-12-08 20:42:59,594 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 28, result: OK
> 2023-12-08 20:42:59,616 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> message with ID: 29, result: OK, transaction: UNAVAILABLE
> 2023-12-08 20:42:59,615 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[31]:NON-RELIABLE:CoreMessage[messageID=31,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@967407085,
> with transaction: null
> 2023-12-08 20:42:59,621 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 31, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:42:59,621 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[32]:NON-RELIABLE:CoreMessage[messageID=32,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179613,_AMQ_ClusterName=ExpiryQueue59d00b93-960a-11ee-b571-7ab0027aea5f]]@1373537114,
> with transaction: null
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 32, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[35]:NON-RELIABLE:CoreMessage[messageID=35,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:59 UTC 2023,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068179614,_AMQ_ClusterName=DLQ59d00b93-960a-11ee-b571-7ab0027aea5f]]@958347658,
> with transaction: null
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 35, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[39]:NON-RELIABLE:CoreMessage[messageID=39,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@333704293,
> with transaction: null
> 2023-12-08 20:42:59,622 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 39, to consumer on address:
> activemq.notifications, queue:
> notif.5fc32ddc-960a-11ee-8014-12a7ed0622c7.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5fc2919b-960a-11ee-8014-12a7ed0622c7, consumerID: 0
> 2023-12-08 20:43:08,447 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=1d1d79ce, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256
> [ID=1d1d79ce<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@51aa6256[ID=1d1d79ce>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:60787]]
> 2023-12-08 20:43:08,464 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=8f88c6d5, clientID=null,
> nodeID=59d00b93-960a-11ee-b571-7ab0027aea5f,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7
> [ID=8f88c6d5<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@14a6e5e7[ID=8f88c6d5>,
> local= /10.44.0.18:61616, remote=/127.0.0.6:40065]]
>
>
> HA “slave” broker log.
>
> 2023-12-08 20:37:58,374 INFO [org.apache.activemq.artemis.core.server]
> AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.28.0 [null]
> started, waiting live to fail before it gets active
>
> 2023-12-08 20:37:59,020 INFO [org.apache.activemq.artemis.core.server]
> AMQ221024: Backup server ActiveMQServerImpl::name=amq-broker is
> synchronized with live server, nodeID=9759f092-9609-11ee-9862-e26c98ea5364.
>
> 2023-12-08 20:38:00,032 INFO [org.apache.activemq.artemis.core.server]
> AMQ221031: backup announced
> 2023-12-08 20:42:27,415 INFO [org.apache.activemq.artemis.core.server]
> AMQ221066: Initiating quorum vote: LiveFailoverQuorumVote
> 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> AMQ221084: Requested 0 quorum votes
> 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> AMQ221083: ignoring quorum vote as max cluster size is 1.
> 2023-12-08 20:42:27,416 INFO [org.apache.activemq.artemis.core.server]
> AMQ221071: Failing over based on quorum vote results.
> 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> AMQ212037: Connection failure to
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> disconnected because of server shutdown [code=DISCONNECTED]
> 2023-12-08 20:42:27,431 WARN [org.apache.activemq.artemis.core.client]
> AMQ212037: Connection failure to
> cray-dvs-mqtt-ss-0.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local/
> 10.44.0.20:61616 has been detected: AMQ219015: The connection was
> disconnected because of server shutdown [code=DISCONNECTED]
> 2023-12-08 20:42:27,434 INFO [org.apache.activemq.artemis.core.server]
> AMQ221037: ActiveMQServerImpl::name=amq-broker to become 'live'
> 2023-12-08 20:42:27,574 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address DLQ supporting [ANYCAST]
> 2023-12-08 20:42:27,575 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> 2023-12-08 20:42:27,578 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> 2023-12-08 20:42:27,611 INFO [org.apache.activemq.artemis.core.server]
> AMQ221007: Server is now live
> 2023-12-08 20:42:27,699 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:1883 for
> protocols [MQTT]
> 2023-12-08 20:42:27,717 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at
> cray-dvs-mqtt-ss-1.cray-dvs-mqtt-hdls-svc.dvs.svc.cluster.local:61616 for
> protocols [CORE]
> 2023-12-08 20:42:51,946 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> 2023-12-08 20:42:56,977 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001: destroyed
> connection: RemotingConnectionImpl [ID=df357d0e, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077
> [ID=df357d0e<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@4cbed077[ID=df357d0e>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:56167]]
> 2023-12-08 20:42:57,443 WARN [org.apache.activemq.artemis.core.client]
> AMQ212004: Failed to connect to server.
> 2023-12-08 20:42:57,445 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 26, result: NO_BINDINGS
> 2023-12-08 20:42:57,447 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6
> 2023-12-08 20:42:57,512 WARN [org.apache.activemq.artemis.core.server]
> AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> failure detected. Unblocking a blocking call that will never get a response
> 2023-12-08 20:42:57,513 WARN [org.apache.activemq.artemis.core.server]
> AMQ224091: Bridge ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008[nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> connector=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> address=, server=ActiveMQServerImpl::name=amq-broker]))
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]] is unable to connect to destination.
> Retrying
> 2023-12-08 20:42:57,629 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=483170f6, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251
> [ID=483170f6<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@3df0e251[ID=483170f6>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:55911]]
> 2023-12-08 20:42:57,630 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=4514a2b3, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b
> [ID=4514a2b3<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@f9b983b[ID=4514a2b3>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:60199]]
> 2023-12-08 20:42:57,648 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created
> connection: RemotingConnectionImpl [ID=fcde4d78, clientID=null,
> nodeID=9759f092-9609-11ee-9862-e26c98ea5364,
> transportConnection=org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8
> [ID=fcde4d78<mailto:transportConnection
> =org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@16ef79c8[ID=fcde4d78>,
> local= /10.40.0.106:61616, remote=/127.0.0.6:53221]]
> 2023-12-08 20:42:57,663 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 29, result: NO_BINDINGS
> 2023-12-08 20:42:57,663 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5ea190dc-960a-11ee-b571-7ab0027aea5f, session connectionID:
> fcde4d78
> 2023-12-08 20:42:57,679 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 31, result: NO_BINDINGS
> 2023-12-08 20:42:57,679 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841002: created
> session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, session connectionID:
> fcde4d78
> 2023-12-08 20:42:57,708 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 33, result: OK
> 2023-12-08 20:42:57,709 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841007: created
> queue:
> QueueImpl[name=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=true]@3645037f
> 2023-12-08 20:42:57,719 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841005: created
> consumer with ID: 0, with session name: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f
> 2023-12-08 20:42:57,720 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 35, result: OK
> 2023-12-08 20:42:57,755 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841009: sent
> message with ID: 36, result: OK, transaction: UNAVAILABLE
> 2023-12-08 20:42:57,753 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[38]:NON-RELIABLE:CoreMessage[messageID=38,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=248,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA=true]]@2125985563,
> with transaction: null
> 2023-12-08 20:42:57,760 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 38, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:57,761 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[41]:NON-RELIABLE:CoreMessage[messageID=41,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=636,properties=TypedProperties[_AMQ_RoutingName=DLQ,_AMQ_Distance=0,_AMQ_Address=DLQ,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=3,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=DLQ9759f092-9609-11ee-9862-e26c98ea5364]]@243253411,
> with transaction: null
> 2023-12-08 20:42:57,761 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 41, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[43]:NON-RELIABLE:CoreMessage[messageID=43,durable=false,userID=null,priority=0,
> timestamp=Fri Dec 08 20:42:57 UTC 2023,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=684,properties=TypedProperties[_AMQ_RoutingName=ExpiryQueue,_AMQ_Distance=0,_AMQ_Address=ExpiryQueue,_AMQ_NotifType=BINDING_ADDED,_AMQ_Binding_ID=7,_AMQ_FilterString=NULL-value,_AMQ_NotifTimestamp=1702068177753,_AMQ_ClusterName=ExpiryQueue9759f092-9609-11ee-9862-e26c98ea5364]]@1544502860,
> with transaction: null
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 43, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841014:
> acknowledged message:
> Reference[44]:NON-RELIABLE:CoreMessage[messageID=44,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,size=266,properties=TypedProperties[_AMQ_RESET_QUEUE_DATA_COMPLETE=true]]@1828250555,
> with transaction: null
> 2023-12-08 20:42:57,762 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841012: delivered
> message with message ID: 44, to consumer on address:
> activemq.notifications, queue:
> notif.5ea5af8e-960a-11ee-b571-7ab0027aea5f.ActiveMQServerImpl_name=amq-broker,
> consumer sessionID: 5ea4ec3d-960a-11ee-b571-7ab0027aea5f, consumerID: 0
> 2023-12-08 20:42:59,607 INFO [org.apache.activemq.artemis.core.server]
> AMQ221027: Bridge ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@2a063f52
> [name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.59d00b93-960a-11ee-b571-7ab0027aea5f,
> postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker],
> temp=false]@737e64a6 targetConnector=ServerLocatorImpl
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@139110008[nodeUUID=9759f092-9609-11ee-9862-e26c98ea5364,
> connector=TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-1-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local,
> address=, server=ActiveMQServerImpl::name=amq-broker]))
> [initialConnectors=[TransportConfiguration(name=artemis,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=cray-dvs-mqtt-ss-0-cray-dvs-mqtt-hdls-svc-dvs-svc-cluster-local],
> discoveryGroupConfiguration=null]] is connected
> 2023-12-08 20:42:59,625 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 46, result: OK
> 2023-12-08 20:42:59,625 INFO
> [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841010: routed
> message with ID: 48, result: OK
> 2023-12-08 20:43:07,437 WARN [org.apache.activemq.artemis.core.server]
> AMQ222186: unable to authorise cluster control: AMQ219016: Connection
> failure detected. Unblocking a blocking call that will never get a response
>
>
>