You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Lino Pereira <li...@redpointglobal.com.INVALID> on 2024/01/09 01:41:58 UTC
Looking for HA/replication boilerplate broker.xml and advice (continued)
Hi Justin,
I’m working with John Lilley on getting an ActiveMQ HA configuration operational. Our current HA config is replication with master and slave instances and with static connections, and it seems to be working, except that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect itself after a failover from the live to the backup instance. Now, the backup process does start processing msgs right away, but the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable. I have included the relevant log traces from the Thread-0 ActiveMQ-client showing the Connection failure detection all the way through when the client is usable again, over 1 minute later. I am hoping that, based on the trace, you can find something that, perhaps, we should be doing differently in the client side?
Here is our Client connection url:
(tcp://dm-activemq-live-svc: 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
Should we also be including the corresponding connector names?
In an earlier email with John Lilley you recommended 2 brokers using “shared storage” instead of “replication”, which I will try next, however, since this issue seems to be on the client-side, I want to make sure our client-side is setup correctly.
Thanks in advance for your help.
Lino
Thread-0 log traces:
2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-live-svc/10.43.197.248:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
---
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616
2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617
2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
---
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616
2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = ced14fbc
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617
2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 10, performing as a failover operation now and recreating objects
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address activemq.management
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536 credits on address activemq.management, needed = 65536, arriving = 0
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536 credits on address activemq.management
2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:212)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1172)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[rg] <https://www.redpointglobal.com/>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Recommendations for HA/shared-store using Azure FileShare for the PV or performant alternatives
Posted by Arthur Naseef <ar...@apache.org>.
Hey John - by "inbox pattern" I mean the following:
1. Application receives a request (e.g. via http endpoint)
2. The request is stored into a table in a database
3. A response is sent back to the requestor that the request has been
successfully received after the DB transaction completes
4. A message is placed on a message queue to initiate processing (this
can be done asynchronously so the requestor doesn't wait for it)
In addition, the following enhances this pattern:
- The database table can be scanned periodically to check for idle
requests, and new messages can be posted to the message queue to continue
processing
- Each processing step uses the database to prevent duplicate processing
and update state when a processing step completes.
The coatcheck pattern is another one that is very valuable - message
queueing works best with small messages (think buying a ton of lumber and
having to push a cart with that entire load through the checkout line at a
lumber store).
Art
On Mon, Jan 29, 2024 at 8:31 AM John Lilley
<jo...@redpointglobal.com.invalid> wrote:
> HI Arthur,
>
> Can you explain what you mean by the "inbox pattern" and how you would
> configure it? I've used what I call the "coatcheck pattern", in which a
> very large message body is written to a database or to cloud storage, and
> the AMQ message simply contains a reference to that blob. It works
> reasonably well to reduce I/O pressure in the broker, but only if message
> size is the limiting factor (as opposed to frequency).
>
> Our experience with AMQ using Azure FileShare and shared-storage was a
> spectacular failure. Under our expected load, which AMQ has no trouble
> keeping up with normally, it would error and timeout when using
> shared-storage on Azure FileShare NFS. It could be that we needed to
> upgrade our FileShare, but it was already "premium" and we were so far off
> the mark it didn't seem worth pursuing.
>
> john
>
> -----Original Message-----
> From: Arthur Naseef <ar...@amlinv.com>
> Sent: Wednesday, January 24, 2024 11:22 AM
> To: users@activemq.apache.org
> Subject: Re: Recommendations for HA/shared-store using Azure FileShare for
> the PV or performant alternatives
>
> *** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ***
>
>
> I haven't personally worked with Azure's solutions, so just adding some
> background here that may help.
>
> NFS performance can vary significantly. For example, initial
> implementations in another major cloud provider were very slow - but after
> significant effort from the could-provider's NFS folks, it has improved
> significantly.
>
> There are many factors that come into play for AMQ + NFS. Typically the
> biggest concern is the "synchronous write" latency - because of messaging
> guarantees provided by the broker for persistent messaging, the broker has
> to wait for persistence to "report back" that writes have completed before
> message production can complete. In the broker, this is implemented via a
> synchronous write to disk.
>
> Ultimately, measuring the NFS timing and working with NFS experts is
> valuable. As a baseline, testing with different types of disks can help to
> narrow down that the performance problem is related to the type of disk.
>
> Keep in mind, I can't tell you whether the Azure NFS solution is causing
> the performance bottleneck, but I can say it would not be a surprise.
>
> A simple "micro-benchmark" program that measures synchronous write
> performance could be a quick way to narrow that down.
>
> BTW, have you considered the "inbox pattern" which pairs a DB with
> messaging? Any place where messaging reliability is a major concern, that
> pattern works well as it uses each technology (DB, messaging) in the way it
> is best suited.
>
> Hope this helps.
>
> Art
>
>
> On Mon, Jan 15, 2024 at 3:27 PM Vilius Šumskas <vi...@rivile.lt>
> wrote:
>
> > Hi,
> >
> >
> >
> > just thought to give my point of view on this.
> >
> >
> >
> > Check if Azure NetApp NFS volume would provide better results for you.
> > This is what we are using for our Artemis HA shared storage cluster on
> > Google Cloud. It is more expensive but in our tests provided much better
> > performance comparing to anything native Azure or GCP file stores give.
> >
> >
> >
> > Also, I don’t want to encourage to drop K8s here, but to tell you the
> > truth we abolished running Artemis on K8s all together. For our use case
> it
> > was just too much work for too little gain. We run native Linux VMs with
> > NetApp NFS attached storage. For the clients cluster is accessible by DNS
> > directly without any additional load balancers. This way we left failover
> > flow purely for the clients and the internal mechanics of the broker
> > without the need to tackle K8s health checks, timeouts, graceful
> shutdowns
> > and similar issues.
> >
> >
> >
> > --
> >
> > Best Regards,
> >
> >
> >
> > Vilius Šumskas
> >
> > Rivile
> >
> > IT manager
> >
> >
> >
> > *From:* Lino Pereira <li...@redpointglobal.com.INVALID>
> > *Sent:* Monday, January 15, 2024 10:10 PM
> > *To:* users@activemq.apache.org
> > *Subject:* Recommendations for HA/shared-store using Azure FileShare for
> > the PV or performant alternatives
> >
> >
> >
> > Hi Justin,
> >
> >
> >
> > Upon testing our “shared-store” solution in our target Azure K8S cluster,
> > we ran into performance issues, seemingly due to the use of an Azure
> > FileShare for the shared-store PV. We are currently using "Azure
> > FileShare NFS on Linux" and NIO for the journaling mode.
> >
> >
> >
> > Azure FileShare info:
> > https://learn.microsoft.com/en-us/azure/storage/files/files-nfs-protocol
> >
> >
> >
> > What are your recommendations on choosing the best type of Azure
> FileShare
> > to achieve the required performance?
> >
> >
> >
> > Should we look at other journaling modes as a solution here?
> >
> >
> >
> > Or, is going back to “replication”, with 2 backup instances to avoid
> > split-brain, the best solution? Note that we do not need to persist
> > messages for any longer than a couple of minutes.
> >
> >
> >
> > Thanks,
> >
> > Lino
> >
> >
> >
> >
> >
> > [image: rg] <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,5ajcMwinNa1VWTcDmA8Mj-nfQOh89Eiudhyxnn4whyIBqIOjlHqNbAhOJdyhfFV1IdcDZPCCvdhfcITATZ9P-w57BqMCkV6a4xW8PThHOaS4qFBBdk0a&typo=1
> >
> >
> > *Lino Pereira *
> >
> > *C++ Developer, Redpoint Global Inc. *
> >
> > 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
> >
> > lino.pereira@redpointglobal.com
> >
> > *From:* Lino Pereira <li...@redpointglobal.com.INVALID>
> > *Sent:* Thursday, January 11, 2024 4:11 PM
> > *To:* users@activemq.apache.org
> > *Subject:* RE: Looking for HA/replication boilerplate broker.xml and
> > advice (continued)
> >
> >
> >
> > **** [Caution] This email is from an external source. Please use caution
> > responding, opening attachments or clicking embedded links. ****
> >
> >
> >
> > Hey Justin,
> >
> >
> >
> > So I am trying to verify our AMQ docker images in the local environment,
> > basically our system runs as simple processes on my local Linux system,
> but
> > the AMQ instances run in docker containers. Now, this works fine for
> > standalone, and for the HA case, the “master” instance works fine with,
> or
> > without, the “backup” instance running. However, the “backup” instance
> > doesn’t seem to ever start up fully, that is, it is missing the
> AMQ221080,
> > AMQ221003, AMQ221007 and AMQ221001 log lines, shown below from the
> > “master” instance. As a result, it never announces itself in “Backup”
> mode
> > as seen when run from k8s. I am using the same broker configs that run
> fine
> > when AMQ instances are run natively (no docker).
> >
> >
> >
> > Also, when I stop the “master” instance, the AMQ-client reports these
> > exceptions and never connects. Any ideas on why this may be happening for
> > the backup, which uses the same exact docker image, just with a different
> > broker.xml?
> >
> > Is there some fundamental thing that I am missing when running 2 AMQ
> > instances from docker containers?
> >
> > My goal is to validate that our AMQ docker images are functional outside
> > of k8s.
> >
> >
> >
> > With regards, to getting you the thread dumps, next on the chopping
> block,
> > can I just attach them to the email, or is there an alternate path to get
> > those to you? Apologize for the dumb questions, but I am new at this.
> >
> >
> >
> > Thanks.
> >
> > Lino
> >
> >
> >
> > AMQ-client exceptions after killing the “master” instance:
> >
> > 2024-01-11T15:47:15.322 [Thread-0 (ActiveMQ-client-global-threads)]
> > client.errorCreatingNettyConnection:741 [] ERROR - AMQ214016: Failed to
> > create netty connection java.lang.IllegalStateException: No
> > ActiveMQChannelHandler has been found while connecting to localhost/
> > 127.0.0.1:61617 from Channel with id = 0bec947c
> >
> > at
> >
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a980&c=E,1,--JOoCrTMX_jWzN7OBNpH51ACSzbmfM2BQMdiUEhw7AetHe2Swr4RH-2iRuMRh98ObfYPZkvyjJz8HJkHoLeGOihsqrgRmds9eKXguMxiiiaKmnkO_OSxj1z&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a980&c=E,1,9sVwvAhg-dCUoWlCTkknJyr35Kb1lORNDVxR6-uqw3mazBhZ5pG19VN92yx8WmUo86gmEfiM-fyes3vKIRRqmUr7xGxTepKfWp8f8YCxIKDeU3MjhJIbTwA,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a866&c=E,1,zt-CVldlmA5WggxivfZazCjgd3_Lj3OY6MGl5OVd2M7PDrI6oaAYX-EmNNsIpeiGNNIjrONSZlgN5G-KK63xVYQ-t6VC6qcluDUpfqgbGDrzAzwfykxIECMWMVc,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a866&c=E,1,PDO7x8ScbX2lkRczVyc7Qx5qFYh8jYI_QZSxIyD15vNJbt-HUSyFIAsL54Aas_i1kj247doxU1D-TF1f6UNtgifP4lOW0jVh-Ca8bmu8X2BRDUZyZYrlZd4,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a848&c=E,1,YgUinkp1CTnxOL4RXehYVUK6aUtheE1ND_UhWmJUJtyKxrilFc8vykkglXCRZUf77BZrtd0U_Ju0a4QgdspRI-oMNyzQh5hWQay4ckS0p4z7Ym0u-xvDhQ,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a848&c=E,1,M5S848Ei6ypG34y6nHCsxBxodJIiZs0WdncK5b_Paenx_yRNXYZToN-BG2GJ4M8V03LsOGQzIKSbFW7uKF8az4WwG4Xrzk0hIvgvf05YYBR-KraWq2Quc6cO&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1212&c=E,1,ZNIZLZvdwYwpL-xkz0pRXdKnZFOsDmCCXzrsdg-EpInUcQLSknJeZu2F_W9AhSILXnxoGmaxlYSp0jqu10gGnm2-Evg3rPp3uQJr8ruuNz4QPaTaKf7rqtOAJfU,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1212&c=E,1,Zu7gFKlm4HmSOURefrcnPu7_omfp3z6NkhUulfWOlvCON5qFl8o_m__R5jhaoNXCjQUWg3afJOkgibeky8zBPzqugdJrnx3mDcRvktMSa8mHPzzyhu9E&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1333&c=E,1,cUJprkrFcHupT11LWCziLhYS20HwCz5C-3yWeqHq3Kdl74IngbR8pUWLi9UlP6laatJYIbCKeGsIaYoYzV7Ii-u_aniwz4JNrK_eH-Bc5ZCx4w,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1333&c=E,1,QLuXIqCAvTMp9vzA3xlqCzu6uu_CAV6-6B3ZjST9QtAqqlHPPW031VktMnXrspcOD4IBb1JKvR2F8OzNZIi97M8NdBmUvk48uDHiFLvxVM4,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1253&c=E,1,tYT4QhUI236EaoHu-bd2NY-uVuJ3WIlZ_JS5Z3hpbZqfxwJ2A2_ki40UOEUt8rtEWNoPDn29f06FV-TbP20llGUE_Jz3rqMh1x5rbWHp99zmVjcgkhg9v9Yq&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1253&c=E,1,HGg4Q1mxe7PWvtXuKXcO1vTG9pGtPTivn_BSZ-cDFd_RB5T6KlxMhp2Z2iKrHRFFrcmAJyClu3xTgdygcYT1cRvfTIU0mnOfoNn73hij&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1496&c=E,1,Xi4IaboIvCsIgJLHmto5_91_7kVWXzu4pT6VOLUB_FckDqFe-MwAf3Tn8wFu6AkYEse2MgOz5PbXcpx1K9ka8jyEaNyac84vtqyeFji3RrKIkupwZsR9xTc,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1496&c=E,1,YDsqwgPbq8QPr_ODsKRUE5-FJdCxEgI2EdJdlE_ogGX_Dtkdqu0ZUW5FnNQHKMwQfJEjscTg0sXGDaY7D27_1M-BNdURAJybIJ0mM63IpSP_Kix0E7dJj0xAAA,,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1074&c=E,1,GyMpBtXF0zJMlPd8R5QcrlIYrG1jYyvHdB-gSpPnT2nKuiKSy0ZA9cH3U9zea6PIldTV6cgJjPRBTTPW65zs8dVfN00ddgEgmMJAtuqEbo2eIxkpyo4_Thbo3f8,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1074&c=E,1,UaHGLwmlvYxEp5Zvya3OgHoGpFPLrtGO8817O18DQ_eCtU9hppVQz3ZSqMOZR4Ou0RyfDnJ25Dsho8qeOiDziqkIZ-8smasmw6AS-SC3C07T9A,,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a959&c=E,1,yTfnN0IvPyn9DmMxZnOeU_hjdrjHupqdbiaqoD9MW1ugd7YWqfKUdlOO3cLcmKnLBPC17TzmKQSyrdmzpZFAvxZwKo8D_KTVwf_XD2SO&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a959&c=E,1,S2vAOnIQrU8itB1DHvtgyBp5ozbblk23_CGr3q-ohZ6Nt35WRISFB1PbqeDDdjRyaj8N9tzZSNUm_TpeEwNQg143NiFeTG18rXYlnq4jakQe2_H0HkGTrg5Qow,,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a697&c=E,1,NTQDf0bLLSrVBiU5yFaEsSR3MzhjCm-v5XL-R8986Qx0bc_gFoW7JJSnOpHti-6JerM4AL-qUl_blORvOvDC6oMrroSKjN1fZh8ipqRK6g,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a697&c=E,1,HRBbQntC7regy9ruiCBdgTxHOjjHiqqEtdrKtyuFoRluNvypXTYG7MHAazXY4rr-7iE9ymvG-ccC_EuBMTBG5AjelMg7KfPvqiIaUkF1RfMIvoeiA8o,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,mQIMsoPaKSQkZ1VRg0ODHorYJuAYyVgvZHMAA239hlc6ujT3xHjKGoOJ9hde44gF5Qlvqo7XPtgCHbhf7Ne99NM2Wf9cnqg0j5ZWA-S4ZNL-FdDAFeDb7dvzkFs,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,WwOwlRDdAEBeA1Gg0ArmZFo1it0EZxIt6RsmaWdNGx6wo_Lz4qKdMxIHy7U_9S5bgl-33cfGzNtlYG9YM9FZXucb7-SjGMsXWWRiyGWMdHRJ&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a559&c=E,1,1IlJV1_pox6fWvlQ5kvv87nkhrf5pk7cRBcMWahs1ElrDscVZtWP-843S1Kjri-B8UPQ17T5oH7AGcB1erP06BjdKHfBhjobugFk8lZd5IcqQ7Q,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a559&c=E,1,Zc7KWsYoeLt5E_P6hgERE60yuujJHXiWpuT9jFjj4gC367dihQ6w7V22qGaDAek1JHPJHKEsYT6vqwQAPm9GSln6px2RaJkIttMBpwOF398rKvWxlzAOmI4,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionException(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a425&c=E,1,vib6Ga2NryGWCm421S_h4Yhos5O7OzRgO5N7LIGZ7BDQQCP5P905eptMajfgBcoR02KRoaVVKVb6a2Rohuqok-qqh66Ev6QY3YHRQBEGCCYXJeCMxA,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a425&c=E,1,BFL6YN5Kxy06vezMnsiL2nC8cdKvlemlxhvuHQ-FuWkoOzxoYzQuft1CeyeiQmEo-L40YbRsfc6dzmYWbWpQQb_xVyaLDbql7YwiFlA60a1xj1PSDmU,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f2.run&c=E,1,7rNv2g-TCyIHjGGhUjbsmsGERf2K_-zsztZJLbyf_SdTRk5n-UMNgBFbJrCJLBdDFKuxzsowV8MgOadI0DUa3rsPmPXvYpOOQF4sYuxTcv4DEE7XStsiH5U,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f2.run&c=E,1,sMyuWkYLp3LX6ibrSEXRcAezwbBM7qAOluPp_GC1jVesMs6jxDzxY4FFM3VjeigvvheAyp3Q7Oe5ACoqft2B-abCTUa6WF1bt7Uh4xjE6hHhWxo,&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a1252&c=E,1,fiJJEVQ9D4XMI5XRdW65UqXquUyRCjYN3ik0ZNbhPCXXCW9ZqpK7XrYloNZKK5EkwjhRbeLdUMF4JeAYSnD4ncjdjnXsIgTq0i-vboU2s_YK&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a1252&c=E,1,3JJSPZ_KDZLXcsGld2w64FmGt132PiKafjM_G1ZXb-ulOdS6Dwv6XUv8QHJqHN6CKkClQ7lUkNnakzibZdks8vAdD3gLbxCOyjzJGO6K745CLD2v2F7UK5zJSQGG&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,2gZklPlcwSiA_VxD57juT50iNxsdC2F7POm135-D7q0KCbqhG2EOqlNhfP57L7pOn4wJqx3QSIs7Ws5LySS4nMJM-6CMVsgE4GYir9A-tRdRUb3HC1bb&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,bpAKAylKI6Drp540buFl0pkNNpNnX4xnMjY3zWwpNyF_392duOqGQPjpdfCPGxrpBvc7_eIyfJpxl4WuufWnNYLFKdIg0Nu9WDi6STLxtiY0tUhT7eeO2iA,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,zz_PQsXSiLhs6hN6dCoMVr__-LNITICfu9whfe7T8sNYKafwJbPx0ydeJIpcOHQlV8KoVCsoJaHdJRvakGUgTlhxoo_r9nBSq-Q3GuOFKT1n&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,CqAtDgvRoDEnuFjXXv-RSPQ-NcQ5-m4IRpyQ9oiN0rhM40sqLFbhY9BMbc0HKKA9gNrRlaFd_DPaSB11LNOv4Rdahsd5fJLOqXHsZnvg3ShSJw,,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,dOtVOpaebNETENjgoNxLOfnEDTl6JXfNUP8nbLUn_ODpPND1BYRdKY7SYb9wd6fnEHv9tKLhtIHJbbpFzgNPaexUYI--qVR5RFenA8fB1w79&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,C5wuD-LRtJ7605p6qERurgxxNLnHBKbC4B7VE9lfDl7f_JsOdDSkeiyMkYACWnhSckLFmjUW2eFKZawLMNYORn4gTtULbgzlBoD8LWUyNto,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,brUqu-rtXT9Jpi9_aPmOt43TyrXnEcWhxFmhTNjRzBGNXk0PNWWIb-ySKwHF38aP2sNKMK6GLK8wLBw8Z-3U2dxR-WQJEij0EknUY1qQKDzdh3Z3&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,rMwMgnKWBZ0Z_t5VCFvBOA7J5McS_4eMRlTXeB6c0yMri_CRxIy8PDWJRjrRtRMI1QpqaQtuBEcG4sCoeTP06BI4NmwaneyL2iZ_EcqSA0sDvg,,&typo=1&ancr_add=1
> >
> > )
> >
> > at java.base/java.util.concurrent.ThreadPoolExecutor$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,0UKW5rRSlatQq443HXmOgPEfCWsEUwzEHwl5acWwEYeN53dHo_aELmkaqio3IDruNjKD-akvwKbWLieB-5SQVagmDJZVxIdMkF9yYvNZjmoTYiaKfQhkErM,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,AmU9mOKxf3Tw98cD6iOKGEUsELr-mALW0vP5N2eC_TxCQIG4d3rxa4xf_W2vXyThIPw3Zmp6MJGpD2c95V3s3JefMOVZJGioR7UrvoKjJHS1Qr22gQ,,&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,m6p9unJlLjm54pezttml4CXXKu86YpKPjtK1wRLPqDvGKX8lb1RF1m2TgYn-nv36VKGzC2Dzm0KhFF2KaR8C-Hw9OQkZslGaEKc2H7DFBDDYMlgRiA,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,uKLItaPZw48N90kGuOJ2giezmgZl49ttEaMZ2H4_Lr1mqtycK7JXSo7FWi90G25OhrIF9k-OxzV8LaDSYfMSQPd4uEksOAIsnz2-SSjTNFctPDccNQ,,&typo=1&ancr_add=1
> >
> > )
> >
> > at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,VSgBOyvoltdkUmo2D1AoPWHOPa-wHjTwCUcCjPpeQdcHexImM4wlFHVXaNKVusUmSbHnK71Af0QNmLJmHeiKW72by76jvIUj4gonFJYK6JyGag,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,gIn498Y42-KQL16Kw7akU8ywkXBbJPMfxg7-Ivrxphmo6NJdJqsbcSdPEAdkwVYS3FgjewkEVfwxLXUtcBsU4QHUbsEaVo5060hl3qEBmZeRmUBspRNTPB0,&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,cSIXdjAwlzhUxm9mRFMDwkvvM6XLO7p6KhGq4WtJI7hrW4tImZ_1CrDPG-ANNyvBMJAlavuv8nLzVesnqH4efC_PzkQF-ceGB8ZGKm9A-BV-&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,KRj2MYUCdG64pKABEvud93N21Qgh8ET58T7SDN4ZzK9ErKl92F_UyzJtq4y9eDIqY04euQg_cRGOTddiPcrePTpRRyy9U7AEk_WKzWUcU_VDwdQacLRCEHF4j6ag&typo=1&ancr_add=1
> >
> > )
> >
> >
> >
> > -------------
> >
> > Log excerpt from AMQ docker “master” instance, showing lines missing from
> > backup:
> >
> > 2024-01-11 22:38:54,844 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221080: Deploying address DLQ supporting [ANYCAST]
> >
> > 2024-01-11 22:38:54,853 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
> >
> > 2024-01-11 22:38:54,905 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
> >
> > 2024-01-11 22:38:54,906 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
> >
> > 2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols
> > [CORE,MQTT,AMQP,HORNETQ,STOMP,OPENWIRE]
> >
> > 2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221007: Server is now live
> >
> > 2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.31.2 [live,
> > nodeID=320ff011-b0d2-11ee-a54c-0242ac110004]
> >
> >
> >
> >
> >
> > [image: rg]
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,WIQnILFOGT12V7T2V5JgdPTF8Y-yZZKUo4gSIgdk9Xb0RKhobbQh-PjrSOCrn9J79hRQkhxWG4u7HGnntjzO4n-TgjsgxAzGPARQB9OM7Zk,&typo=1
> >
> >
> > *Lino Pereira *
> >
> > *C++ Developer, Redpoint Global Inc. *
> >
> > 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
> >
> > lino.pereira@redpointglobal.com
> >
> > *From:* Justin Bertram <jb...@apache.org>
> > *Sent:* Wednesday, January 10, 2024 6:57 PM
> > *To:* users@activemq.apache.org
> > *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> > advice (continued)
> >
> >
> >
> > **** [Caution] This email is from an external source. Please use caution
> > responding, opening attachments or clicking embedded links. ****
> >
> >
> >
> > > ...I’ll have to instrument the code for that, right? Or am I missing an
> > easier way to generate those on-demand?
> >
> >
> >
> > I'm not clear on why you'd need to instrument the code to get thread
> > dumps. You can generate thread dumps through a number of different
> > commands, e.g.:
> >
> >
> >
> > - jstack
> >
> > - jcmd
> >
> > - kill -3
> >
> >
> >
> > You can execute these administratively or just set up a script that
> > generates them every so often.
> >
> >
> >
> >
> >
> > Justin
> >
> >
> >
> > On Wed, Jan 10, 2024 at 7:43 PM Lino Pereira <
> > lino.pereira@redpointglobal.com.invalid> wrote:
> >
> > Hey Justin,
> >
> > I did finally get a chance to run everything locally, and it all works
> > smoothly with failovers in both directions and the AMQ-client recovers
> > within 2-3 seconds and with a lot less log noise.
> >
> >
> >
> > I did not get the thread dumps yet because I’ll have to instrument the
> > code for that, right? Or am I missing an easier way to generate those
> > on-demand? This is happening in a k8s pod, so not easy to connect with
> > jconsole.
> >
> >
> >
> > Thanks,
> >
> > Lino
> >
> >
> >
> >
> >
> > [image: rg]
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,P1-JTENl1pkpwfayJeVxTcbmyxoXqv5aWc8dgoVoQa2P9YJ6ITYanSwPPWI3OOvd-1ZuyvCfDdUgZSHNKSzXPNMT51ZJGff0MJI1JNYM4KnicxOXm0JmVYcKXz0s&typo=1
> >
> >
> > *Lino Pereira *
> >
> > *C++ Developer, Redpoint Global Inc. *
> >
> > 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
> >
> > lino.pereira@redpointglobal.com
> >
> > *From:* Justin Bertram <jb...@apache.org>
> > *Sent:* Wednesday, January 10, 2024 3:02 PM
> > *To:* users@activemq.apache.org
> > *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> > advice (continued)
> >
> >
> >
> > **** [Caution] This email is from an external source. Please use caution
> > responding, opening attachments or clicking embedded links. ****
> >
> >
> >
> > > So our issue seems to be only for failover from live-to-backup,
> > consistently.
> >
> >
> >
> > As far as I know the client doesn't make any distinction between these
> two
> > cases so I'm guessing there's some kind of environmental difference
> that's
> > making an impact.
> >
> >
> >
> > > Is there something that we are perhaps not doing on the client?
> >
> >
> >
> > I don't think so. The connection URL you provided previously looks good.
> >
> >
> >
> > > I will extend one of the example codes to use threads and sone of the
> > AMQ-client wrapper code in an effort to isolate this behavior on a
> simpler
> > code base, as you recommented.
> >
> >
> >
> > Before going down that road it might be better to inspect the ostensibly
> > low-hanging fruit first. Did you happen to test everything locally? If
> so,
> > what happened? Did you acquire any thread dumps during the delay as
> > previously requested?
> >
> >
> >
> >
> >
> > Justin
> >
> >
> >
> > On Wed, Jan 10, 2024 at 3:04 PM Lino Pereira <
> > lino.pereira@redpointglobal.com.invalid> wrote:
> >
> > Hey Justin,
> >
> >
> >
> > After starting fresh with version 2.31.2, the examples work out of the
> box
> > for me as well.
> >
> > And, after ensuring that version 2.31.2 is running everywhere, client was
> > still using 2.30.0, and refactoring to using the recommended shared-store
> > HA, our AMQ-client is still behaving as before. That is, failover from
> > live-to-backup, client recovery takes over 1 minute, and logs report
> what I
> > posted earlier. However, failover from backup-to-live, client recovers
> > within 5 seconds, which is the expected behavior, and our system is
> happy.
> > Please see new log excerpts with more logging included for the 2 relevant
> > threads.
> >
> >
> >
> > So our issue seems to be only for failover from live-to-backup,
> > consistently. Is there something that we are perhaps not doing on the
> > client?
> >
> >
> >
> > I will extend one of the example codes to use threads and sone of the
> > AMQ-client wrapper code in an effort to isolate this behavior on a
> simpler
> > code base, as you recommented.
> >
> >
> >
> > In the meantime, I thought I’d ping you in case you can think of
> something
> > obvious that we might have missed, especially since we just moved from
> > 2.30.0.
> >
> >
> >
> > Thanks again,
> >
> > Lino
> >
> >
> >
> > Our versions:
> >
> > ARTEMIS_VERSION=2.31.2
> >
> > JMX_VERSION=0.18.0
> >
> >
> >
> > Logs from failover from backup-to-live:
> >
> > 2024-01-10T19:52:29.588 [Thread-24 (ActiveMQ-client-global-threads)]
> > client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> > failure to dm-activemq-backup-svc/10.43.229.161:61617 has been detected:
> > AMQ219015: The connection was disconnected because of server shutdown
> > [code=DISCONNECTED]
> >
> > 2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> > captured on connectionID=d7ff7885, performing failover or reconnection
> now
> > ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015:
> The
> > connection was disconnected because of server shutdown]
> >
> > at
> > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,Mr3pbdt4mcmFaD997_pq47Opt2EcMvLLrsHvGYc39vqgE1v31obNH7GgtocQkwJ8I7NdstAMkov-HWYt3HH43x2PfQNR-mmb8DU84BffgA,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,O0LJ4VfRCemWEaMLULajHsgUo6YseAVDyBlrTm2G50vM20WhlTUV0ueY1DxMRAMStXAoc6wy5Xx3x1vi2XsnE5F7_iNbL-MZ03M4IS9G&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,5uxOr5VVLMHc0sdf8gtHE4s7BtRHg-Mv9J8MsQZwLB9fZHgQIL9-gCiBOnBZ4AFw5etlj6zKObFUYD4UDlTyxJ_Nnsh0-ZD_zKuNa9TYGWXoDBciCXBTXfw,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,VdcejKqF9B67eWPDv-PP_0gR6PEmUxFYHHtW8kyn3Gzg3q-HHzL-Lr7oztjj2LQumPJWO-1KF4OPavU8c3DW4zPgrUsidhnnCpoMUUn_&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,esKwX1OYT2CmcBEJf13EtgEtM5bjfJWvy3_kNngn3FevdhJTot82KGD27QyADUHosT5nnuoY68T0KvZtSMshB8CtXbEo4iuodyuAe6run-Yf_vtpdfZb6iY_jw,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,xKCDuAN84vJBjsCp75epCz7V1lR05TFhETdvpPcKntBTPnQk0zuuQSei7-ly1RPUUsHDP8ZAe-IevqwGtTGfEB_yNzw90QcOpEx5BaXOW6E,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,Em77Llz_l0_fpFOmT3j_K8dNPtK-V-hg3vcuhVhKANwZIdL4YZBxXt_ZqdNLQ9Wjl3LjXF4dODyvIovi7TT1_YjfHVyf0-7hNALJSTea4D01F2WNgy2JQURVGzkZ&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,pq7BenEb7b9i-FYz1LnTy-gHYDdV-28FiSv3pkV_i-1TQZtRxakbT-rOKkgLgz2IDx875Dk5EE_OrwsUcew9TIsvrcZZZ9ZIlwboxMEq9xnR5odpQL8,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,mtE6QufgedGCjjeJ52aBLxixFnx_MdeBZW_SEsMGLo69xglp7tW1gUctbMB7puEEJQQ343WtJ5VXehpalJB8CbzpSMkL8xBDtis2B56Vr3hkehfBrl5Y&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,8mmrZ5alRiVlyEshqjlGpedpt7TuUVBJOy8DDm2qHLGoY6v3pR76llV59sMsYwy2v4rwLqiD9Rcf3Hg4QSCZo7rdO1rQgMNapCKji2jg7T4IJOk3OAeLnIs,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,ASjiJy0t1SbKglekpcLqYuAzRZ7xPKk6a04qfGkUnOjLxrlSWELC0E08z70MOAVeMF06IjjCFZSpnFMLTpZ-14OMPdIRtZymzcHO9TnxVWm3fyrpiQ,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,RLTYzCoyknTJ1GB9szMq3Jg02QAnt7juedv1HCTYmmA4JU1DO6MlJhkvzOTBqHbeeOdKSvwkualBrhGt0fUk7Nl5YLzlgfepqTWcDiBmXQ-_xSMcAHPw&typo=1&ancr_add=1
> >
> > )
> >
> > at java.base/java.util.concurrent.ThreadPoolExecutor$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,iy0sncbqed4Eoa37FJyaBMZ1dBdf4bdmG0oRJM4-hoomUQiyzi49VCEaJhIjYL72cmZ4An1QpH5PvNS-fTen0fShUt4Ft9eGa7K36Axs9d4,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,iYtEGEWoGf58Vq3CGAxBrH3iWZ1jWm1N1b3AOFPNlx0kCahcxHEwXT7ZAQ_h3ynXTeKJSkvmBiGUkbqE3bH7tGw-Gn8VV7R7LaCkxl9Dch5I4kQDfJDKSg,,&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,FfU9MXaLianm9R_iutvVZro3GOENZby33tX7xvKXuWHPcjQQFRE3AH6Jn59gGGRXgpSqQZj44nrnkXXiZuJF8dyFHp2yO0m_E67lXB2wCQ6NFKE,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,kH54RADAl8RcHDeCxJW9b-P4YJJFElj_Nbh6mHtmwlFdP2tKzzbK3wkAXg9tVqhIlsbd4gr2LJkn7DWZ4I-J8WVE-7UovVbVakuQTTxc2pbR9vuIDiEFztc,&typo=1&ancr_add=1
> >
> > )
> >
> > at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,20ZrvQagCRF4QTYRrJ6tsdMTwZ5hQ2XtPVPpdtR3XEK9tni5a5bBz72Os4-2LPRdgsQssQTFjXaSvbIyReL98hIBq0LJ6Xg6u6NQO-VqpG0S&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,V9-XUYgB7GoqPyGhEdE_CmtNood-yvXw6P9wRIqQOSYSWuTfiF1bhonQejBQfgJdHq8k2TsE1RAvcSOp_WzBE-ybl0l6rc5pIuMGdC1LUITpdk_Iuha7urSje4Y,&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,86MdH7Ho2updjk9VNoTbiFAZDkPZva_rcFIo_FQMkfNJuukTUa1ho-6FJSKBkrSDldwQIHL-7NAcbMjdrMtoIyL3JwtZRZLRtf-t1Sw7&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,RNgnFH7dBkD27TX353Mc-kqneOLcqFXEgpGywxg8VduTTYNyYFt5VqkKfirozQw9V88WOI8XK4v9dvrtPIl-3u7cp673GvKxq3L-XgqypuDtSnkRIYnh813U&typo=1&ancr_add=1
> >
> > )
> >
> >
> >
> > 2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> > reconnection attempt 0/200
> >
> > 2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)]
> > client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> > failure to dm-activemq-backup-svc/10.43.229.161:61617 has been detected:
> > AMQ219015: The connection was disconnected because of server shutdown
> > [code=DISCONNECTED]
> >
> > 2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> > captured on connectionID=d43c72b6, performing failover or reconnection
> now
> > ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015:
> The
> > connection was disconnected because of server shutdown]
> >
> > at
> > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,Pb4Gjw8cAEK4mRPSKvySF-TZaU1MIQLpfPuBIgcm6dUgwnlv7IrUo7JZPV5HWrp50EnTgrQLowfSia4dVSwM1yYk6SWvd6O-FGU-JTKn3G-XpQlw&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,L97BTn6AFLbUpYgHzAlQm9pDJ4Sn--TlGDpz05xOE3xd-qn9krfLJ6JzGTaiSpQwuVjuQ8KUPfUSW6N18pgw44-ay7eqYb2h0p3sxehZeXAC&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,SFBqyVG7Ebud5QIGYSfD2X8qDRvMZ2KUna2iM2x5eti-y7lgMHVrQZBEQFwuW5LKaFuAc9Qy3LSF2WfNwCoavbGXEnp-fsqZmlWmk_bojWc,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,PiedcDlDv-nxD-EF13k9j-sagD4clTOVvCs6xyDQvnrPzDUxc4xAbnHg0d7hnqeG4cVJZaO42LJvoNm59E3SmClgLAft-3N7F898HO6Obbkm&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,iAzsBKgfMT7O0K2WksIFtRro8pwHur_nVf2Mxxf7qb9X2As35BL5LNHfI7mq4bj-hNtMbAUEmNWso5KnY69nZhYm7w1g9cUtuHY7o6wNJIRaF3Y,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,G5SZ7tGYpGBQzNRxdmIzsp4hWlUsmbY3_881k0N5lFoFBwuHt1dMCkKHI8wiZRGpS6qDY_UDBvDmKVQNRIHXIIyg5Df2nnWDCTLnbvWCO20fW0PC&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,9IAw9lo-op4sAvrd7fWnIK52Kj6kbL1vJXQDSwfvAFWwNnWVzRXRVamDdm8qtwAyEqxehrhyhsfWcx_l8vRA3RZ2sdPZlX4FYOGgLd73B6GWq71t1BKwm1c,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,MsagygznR7ie9qgJLlMogXq8ex-X4ST-fg_vbVbgcBanXur80DVykS1ftnzq0e4bWLE0k8EiNmlhlezemqTlBEncWMJz-AQAKdTxEk4WyVY,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,uWuvrZ5S1eIewMe0I_OPx1KRnyK_EFNo7e-8kD4KbrQEveKYerYlrcyTZRsU8tiZvDCsSC6cs02SBC7v-barc3wMFibDjfG1DSPeiot8HHozgiQ,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,n7l9CUpZZbFH3ca7Iu2wxM3kNfged80mMRcu0Vn9wIoZz7aKzCP3c1OVR73iqF0oLbir7CjMHKAu3sfrR2cLs-EP7orqDWQKQApkzTZra9CJag,,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,WquhQ2ZrbyUtmzRk9zvDtleE6eDakAawIG_S7GMWeknBD-w9ubvsR4JenMxQRC_LRIHmwoDFk69sYRfGdWud872mwzcasuNvSRpohGjF&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,PePvk640nUsvwJWNJY39J2R954rSgiFVPPSQLOsjYskMxtDnDVBuzQiZ8ut5SAPoqzV9uYW6GX3ekOAAoou9eu6VpHa_oFc2-7d27W00VBLU_o_Zv4wEUw,,&typo=1&ancr_add=1
> >
> > )
> >
> > at java.base/java.util.concurrent.ThreadPoolExecutor$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,WKXDaNmnUkiw6yQLU8DeLmCsUKQ02DYgVbiKdQlUJVDtysFCse7EIVda1biRsHIMR7fSPKb425g5EMmgZg_WqixOfOrCF4AXk3iBNo2kL0k4pTOcB4phWg,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,REmfZe55ytZRsqZvFkCNsYUcRO5Yd8PZwIlQfGystV0qUeF2mXSYaWKgleI098Ac40PND4ilF-ukqEWDY8N04AIRQgGgKGpG036xx7jNSc78YS6UbKWiMNUHfNc,&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1_QKSgioBO_0oJhMZNHydhglIxieW-0-3PKhoKaGwfuEJEpPYoUGcTom7KYsAwd9SKfzJvpaHOV7mLeBqqEIz2jc5rhSvLMO_K1wHuRr&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,nnqfN76WKNs0ZYhVPzea17QO5aWkwmLqbfYrmQJ_qGQIgTzCL4PSXhjU9jy4mhbCUfl53cwF4ISYabr_zD1GUvCgv5m4dYW1_OROCBpe&typo=1&ancr_add=1
> >
> > )
> >
> > at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-elXqIfEcoGJrq2i3bA_kciC1k1tdjh_d4QwvDsIGDqWWE90FQXTc6Auo7KAvxz35dJB0xa76aEDYpl2pOm3UjommGmjsTEoJaLHASKfMw,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,vlJ5vYZAfrlVp9kL50a_jwWhqVW8-LMUgzeBb6mn-udcS_fMR1yld889bXgOXTMDiutFqKYX9XPh0RtxeXRLtnN31WqeOlH_CZv4_ZyhnOwXvm4J&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,6vJ101x4RKUCOktPDq3GW-EUpGhMX8p7F8zG5XtKtQWXbkL0x3IcEVLTzeALD7hlP83H_AoEGKTHtcsMiJ7PJh-GGfjxoXx5-EdWXbVHBLOE4w,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7Ba-VAkMm3x0E0f6F1f73TH_y9Ylo03hx9DEz4kqWzvrXy0gAv3D9zIM35eXIaB95UFxo72DkMpb0nKW_a-BIF89JobbRqjyVNLdC5K3L1EM520B&typo=1&ancr_add=1
> >
> > )
> >
> >
> >
> > 2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> > reconnection attempt 0/200
> >
> > 2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> > and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61617&host=dm-activemq-backup-svc
> >
> > 2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-backup-svc:61617
> >
> > 2024-01-10T19:52:29.592 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> > and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61617&host=dm-activemq-backup-svc
> >
> > 2024-01-10T19:52:29.594 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-backup-svc/10.43.229.161:61617
> >
> > 2024-01-10T19:52:29.593 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-backup-svc:61617
> >
> > 2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-backup-svc/10.43.229.161:61617
> >
> > 2024-01-10T19:52:29.597 [Thread-6 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = d4cf19d5
> >
> > 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> > and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=dm-activemq-live-svc
> >
> > 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:29.600 [Thread-5 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = 7b2392ac
> >
> > 2024-01-10T19:52:29.602 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:29.602 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> > and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=dm-activemq-live-svc
> >
> > 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:29.603 [Thread-7 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = c1fb4c82
> >
> > 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> > and initialConnectorConfig: TransportConfiguration(name=null,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
> >
> > 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:29.610 [Thread-8 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = 06b7b3a3
> >
> > 2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> > connection been made, returning null
> >
> > 2024-01-10T19:52:29.611 [Thread-9 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = 65680be4
> >
> > 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> > and previousConnectorConfig: TransportConfiguration(name=null,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
> >
> > 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:29.624 [Thread-11 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = 97a109b7
> >
> > 2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> > and initialConnectorConfig: TransportConfiguration(name=null,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
> >
> > 2024-01-10T19:52:29.625 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:29.629 [Thread-1 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = 77b2de2f
> >
> > 2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> > connection been made, returning null
> >
> >
> >
> > 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> > reconnection attempt 1/200
> >
> > 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> > and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61617&host=dm-activemq-backup-svc
> >
> > 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-backup-svc:61617
> >
> > 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-backup-svc/10.43.229.161:61617
> >
> > 2024-01-10T19:52:30.612 [Thread-3 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = f92d7cd6
> >
> > 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> > and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=dm-activemq-live-svc
> >
> > 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:30.613 [Thread-5 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = 1d44895d
> >
> > 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> > and initialConnectorConfig: TransportConfiguration(name=null,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
> >
> > 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:30.614 [Thread-7 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = e8602acd
> >
> > 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> > connection been made, returning null
> >
> > 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> > reconnection attempt 1/200
> >
> > 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> > and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61617&host=dm-activemq-backup-svc
> >
> > 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-backup-svc:61617
> >
> > 2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-backup-svc/10.43.229.161:61617
> >
> > 2024-01-10T19:52:30.631 [Thread-9 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = 0a217bfd
> >
> > 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> > and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=dm-activemq-live-svc
> >
> > 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:30.632 [Thread-11 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = d2aa00d7
> >
> > 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> > and previousConnectorConfig: TransportConfiguration(name=null,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
> >
> > 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:30.633 [Thread-1 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = 9a4f639e
> >
> > 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> > and initialConnectorConfig: TransportConfiguration(name=null,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
> >
> > 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:30.634 [Thread-3 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = ecdb1cbf
> >
> > 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> > connection been made, returning null
> >
> > 2024-01-10T19:52:31.000 [Schedule-Runner]
> > SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service
> > as system
> >
> > 2024-01-10T19:52:32.001 [Schedule-Runner]
> > SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service
> > as system
> >
> > 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> > reconnection attempt 2/200
> >
> > 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> > and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61617&host=dm-activemq-backup-svc
> >
> > 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-backup-svc:61617
> >
> > 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-backup-svc/10.43.229.161:61617
> >
> > 2024-01-10T19:52:32.116 [Thread-5 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = a6b3d488
> >
> > 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> > and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=dm-activemq-live-svc
> >
> > 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:32.117 [Thread-7 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = c0627cd9
> >
> > 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> > and initialConnectorConfig: TransportConfiguration(name=null,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
> >
> > 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:32.118 [Thread-9 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = 71523eb3
> >
> > 2024-01-10T19:52:32.119 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:32.120 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> > connection been made, returning null
> >
> > 2024-01-10T19:52:32.134 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> > reconnection attempt 2/200
> >
> > 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> > and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61617&host=dm-activemq-backup-svc
> >
> > 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-backup-svc:61617
> >
> > 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-backup-svc/10.43.229.161:61617
> >
> > 2024-01-10T19:52:32.135 [Thread-11 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = 68efd0dc
> >
> > 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> > and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=dm-activemq-live-svc
> >
> > 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:32.137 [Thread-1 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = daddef73
> >
> > 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> > and previousConnectorConfig: TransportConfiguration(name=null,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
> >
> > 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:32.137 [Thread-3 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = 84813ce3
> >
> > 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> > and initialConnectorConfig: TransportConfiguration(name=null,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
> >
> > 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:32.138 [Thread-5 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = 533990e9
> >
> > 2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> > connection been made, returning null
> >
> >
> >
> > 2024-01-10T19:52:34.370 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> > reconnection attempt 3/200
> >
> > 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> > and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61617&host=dm-activemq-backup-svc
> >
> > 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-backup-svc:61617
> >
> > 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-backup-svc/10.43.229.161:61617
> >
> > 2024-01-10T19:52:34.371 [Thread-7 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = 1d8df71b
> >
> > 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> > and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=dm-activemq-live-svc
> >
> > 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:34.373 [Thread-9 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = de20e193
> >
> > 2024-01-10T19:52:34.373 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> > Connected with the
> > backupConnectorConfig=TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=dm-activemq-live-svc
> >
> > 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> > reconnection attempt 3/200
> >
> > 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> > and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61617&host=dm-activemq-backup-svc
> >
> > 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-backup-svc:61617
> >
> > 2024-01-10T19:52:34.390 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-backup-svc/10.43.229.161:61617
> >
> > 2024-01-10T19:52:34.390 [Thread-10 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = 4661fd5d
> >
> > 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> > and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=dm-activemq-live-svc
> >
> > 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.45.150:61616
> >
> > 2024-01-10T19:52:34.392 [Thread-11 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = d1c14a74
> >
> > 2024-01-10T19:52:34.392 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> > Connected with the
> > backupConnectorConfig=TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=dm-activemq-live-svc
> >
> > 2024-01-10T19:52:34.404 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG -
> Reconnection
> > successful
> >
> > 2024-01-10T19:52:34.405 [Thread-9 (ActiveMQ-client-netty-threads)]
> > ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up
> > backup config = TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61617&host=dm-activemq-backup-svc for live =
> > TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=dm-activemq-live-svc
> >
> > 2024-01-10T19:52:34.405 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 38, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.409 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG -
> Reconnection
> > successful
> >
> > 2024-01-10T19:52:34.410 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 37, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.411 [Thread-24 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 10, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.411 [Thread-11 (ActiveMQ-client-netty-threads)]
> > ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up
> > backup config = TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61617&host=dm-activemq-backup-svc for live =
> > TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61616&host=dm-activemq-live-svc
> >
> > 2024-01-10T19:52:34.413 [Thread-24 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 11, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,PTh4iDktbimD61S_2u93Qof0SOatHLPxYuhTwx4RmztxIbGXEw4EVaQT1sQ5UdxKH-SEHGoaBTBlboJQju_O44SFZvaaWLO4-MmXleNiYsXze4oL&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,S5999ok8Om6JjNfBLhC0R2hYdl3dGqPWsLpis0MTx38AhKXPNqMMlH41RrDz0UZsxZzSZbLvUEOYFEa7hwM4N7gAZfCS9udcqSmOwhLg&typo=1&ancr_add=1
> >
> >
> > 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,GIwgBhyL-Wik9l2AqNI4MjES1gdmJUfEqVrbtU3wLgBtGl0NlhyMc7o4-4Kl09iahBVAsbPUdhwYPs-RPeg5-ZzPPYTCAeU3rPM9jvmXg_xqUP9AeLsu8AXKnw,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,CF5ZPcR0aBH6uaRjLjSeWoBKLnRwUOOgvrD5MzWuerKGIs1FEa5vZF8rdF1kWSKbLwGQlm1IsOeyey0f1t7eiA1ApUgs1EDhL7ilSV0c6rFxEanucXI,&typo=1&ancr_add=1
> >,
> > needed = 65536, arriving = 0
> >
> > 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,h6ooxFeoDRKOe6DMrvyxN9DvLJ4nTkX8-TCIoHrc-Xf9WcSsal_9ZcG5nby3OeHUxsCLiFlHvYoiGBCvFWFPVCT9MCY89GTMxBPVt6ikH_tq5aCUbw-Af3P5Eof7&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,nl9VlpvL5Lx7cxWUON3eMxxXV1my7fPq5WQLaWD1ozszhEPr1wVfqCwFilScj02MzRACloIhLZ8CgyMWoa_OqCu-Tf4folOdvKvtooRq&typo=1&ancr_add=1
> >
> >
> > 2024-01-10T19:52:34.415 [Thread-24 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> > captured on connectionID=d7ff7885, performing failover or reconnection
> now
> > ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> > Channel disconnected]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,heQgfr1DTWeXz6rPEBmlpzRARyTcEfhTUHsO4tP_oqHC0157GZ10xCh6kckF7RiFAWYwhntDKaSjR6taVznn4QO1BxPmGpIzx9XNW_Y5CrSGq3kF2Spq0KadplA,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,awSP7U_0_MIm7Km9lwe59SrlpKsd9iUMA2y266CCeNZS6GJuEExdDYVS6EVMnlmPlVN0W1aqSx2Rss2Dm_YIM71s5j7OAUz6pAdGaTqnU_h2RyaoD6KyuoU,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,sNDACy5rBtvF3taiFbSF6Lwj-3S9xNTRGgz8biYJnUVv9tKyCw-XOFDaeS3QCKCwEskbKsJK3TsvKwTjqKClYruLL3tdaQ7Q95FhBtt9cbjs8gI,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,8gAKfG0MZn0OMdhCHJ98evOezuGoCS0mNbGWplSwawxhF0ed3tPq9vKWoWUbn2xmoLJ34skkCTpI2tUuevSCeVNT00tNpFlOJlyAjGWu&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,OmCd-aJ9EwCguVqoTqWVTAotLX0mbF8oT3l8FWGQ5PdepUgkFuHDli-CBGY_bflWpOE4vCbvOC4wVmR6NSSH-SW7eDc29yrDo8pqWHwqE5AByEQ,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,tXm2hjddgUkJac0QpMg4nVt2gwg5e3ORdql8K6xsJlBZIxph-NuPt1Ks2iB0ZarzoFELPAuAerVMiQpEU3nkq1JiqLAju26hE6Y2c3MwmKRTj66O8tsfQ-WFSh4d&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,T33i42EbsHkh2YDdE_TUPZeZVywUqlNZnmvDGLR9afQz06e5CEMug1KrQHqXcgwdVbJwb3N1U6pP9VW_c72K7vptWyoU95F4RuMxp8CI-ypOIw,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,P0xsXGcezpL-KzlLSOt_rfr6b6RF51B2hthJzXMGD04QeyLyg6AnMvF4M8ZOEwF90ojRdT3CtOy0x9HBSpGK_mmF_6Zl7Muz2TKrZRLAxQRxhV5XnvYlwQ,,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,EkDWiSpN8nJrODSKBaJDEOcA-o_s5mt4eZAW-Yye2VQRh8fil0VD85DfVJKvjfhNZ8mbuniJBnUkN5CyozCceb4vFqbrM3KQ6brg_Kg64w,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,qctWvKkM1LAyegiROEl3hsiEN3dPcBcH4jlkBConOP-G2cgGl6X6XG-kdeQuBBOC8BmtTf-VmlbpJTjC_wznn8I6g7pLOg1uof2v0j_r3yyK7U8,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,Ls7Rscoa5F2LNCLQqRNaI79Grf_Nj8W0C5zHYWcLXmlhwHE7D0qaQpynpuVb41nbC88w9A-BIDeTNWxqKUjhHXAIyykw4kNA1OjyFnMGiAXv&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,KRtNFryEe6jhqdT3of2FZC9PF7zGlYTxJ3KPFwO6TViZkyaHiybwHP3JmOhQ4F7yZtorUVRe5phyCswDWLc8lIFBVowIJygjwxLmO-F3xJSonSQ,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,wTUvUO1DE5J8AJnZBpatxMFX27qqm-i69bNBA8umuqKFLjBUsYeoJRa5hda8r_tsIBFQIJVLhus2JZPlBoDWBiq9yEqsK9P0z8B-AN1Bw8HMFeZy&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,ltH-LxKKSiKev4cs4CgS53B1Sn1z2t9IkxY_vjCKeoXMalJfF0yTujWPogG1a9qkWV5DrAT_qCTk7MdQa3LpkAae5yHTe_19fREF16p6vMMJqGXqLjw,&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,U2u9N3XZe3KrRAFV8FC6A0jb-i4WBpO8GabyKN3J_ODIs03SMJ7qyEFVS7_JKTUE3DJzdoAE8Xe54QzWW_zirqAVS2X1Bdg-Jgb0WJY72DgHw7U6zuGETNTb-oo,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,66enqLb5-TVz_cB0ZCkDW4SpX4c3v0jslINMsxhHSjXfBWrIFoM8PxfSS7_VWmc_MdfmjPHIKcLm4zfcCe4MggwEWLxB-RjiUMqnmc8ShDJLoit5&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,8HE4vq7eDvOR4Akgo1D2PL10SpipE8h_Y8GjSHFCP5Qde-w7osTJWj6gmORajGL_vaB_k321Nub6ghLN08IpaSQY3kUUISFTpTQtIL_sWVfg1z2nup521pY,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,xM_aKOMjg39ORE7gTvaGKP72ytDNvt6tjanfNTzekduYVGBvn963oijBiSY1JOQeorfgcCf57lVZSYIQ20x8sOFAH5zNexuTI0rBtrkDrtcJpq-S&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,8KL2V5Hl_UJmFqzePqxt1m3TWIAHVbzvkIeRkpBBo4pbg8otQlW1R42WImcT3U7DhvBAif2arGKYqt2zg8k_ife9D8XXoyhw1HWfRlp8IoKyDxSJwKzrLQ,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,Jr0OyBaxbJX-r7nzGvHtUkNxDkYucNHeuCPW_Mqw5I4IS0dve_XCjayhv7uvFQTdinHTsHqf2ZtZAn_qz1ZnE04PEsE0KWGfHQUj6N5WIKcNhyQ,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,ZPAthbXOO1oRXumH4hVlQm7THB9GXz7u3roG6FE_qLs1OzYIC3kGaWF0EqXlBbqhaVgSpZtFqo4zgsGHTD5ZMzLA4uhu5UijndwmxkwE4P1qqBX0A1rpTm1hAw,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,fTod1kMxyNrLe6wmpPVQB1vco-7IrNIs9Dx8h9kWYDCH7xYH8V5kj2iao_X4D62DX5S24Xi08LA-xy2zHwj-LIzSKOA38iBBAlHdMD7qu3lK&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,iYpsZX-jjnbMrsS1VvCBeXnN4ZoC31VLIPM4c1i_l5RmG_0_FTxKLBOSY36ZxeAHJYyQ2hjrtS6gPKfbYmS3x0upQbuFRCpQQ54oW5CVMcUeaDZKblgP&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,37VGZcLM1KBT3tttPbLrhK822id0s0HCTimMYb18Jk-8bCbWxvPl2N9_yFFLgXa5pC9DXbKX6c8DMHolUFme8KRWVQdBx_lbRwtpLIOp6O7uiuV0fttkzvI3jxdO&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,5BW-BLxrERkrSISvOqkdNRbxf6b8De8KYd0krQIONwx_IQ5j0CQKg-8wJJAEVvmiADESxORcbca8kwQbvkQ9gCUsFpaT5SXFs67USI9Bj9g2BWJJ&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,zIgfFXrrGxOB_lKTtscVc_jXQlq_pH7TMnaK0IkixM4MVqoSgc-Wp5VV-QGQWmUtW5dggcgtYpRa7OoRHCBvDew7fxYJq28eSZVguCiuBiH9DzdKTHAXC5o,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,RBf0UlvPRwHDakldKITeDilMpRb7wrv829yGFB0b1HT4-n1J88dG7WMGe3t1RooyQQ3kGGqr7TNJmm9Of-ySWBfkV33mxvs-ZgzZB_eIScsIEQv6eO07&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,b6YutPqRZaL3jtcBxfqj_SiU4M_x_Qx-GeSmBfL7c6CG24IUL7AdjJxpyosXZtCtKco4bSDRxOj7HJn5CWgbgbPzJ0TFuXzJjs7VRUKVhg,,&typo=1&ancr_add=1
> >
> > )
> >
> > at java.base/java.util.concurrent.ThreadPoolExecutor$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,3L8S6-PUyDPeFVGKFKyVjd26lXnTihav9ECN0TZYTsG8KgSV9bRAVwWAJZMpzImx9pHWaIYywf6dJN7oqN0-Dr2iLzUutZeHeC4dyhh0StMIshBrIU8,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,xdw7r1BqGJuJPUUyo2a9wj_y2KKTC-j8c0DAocpHKzpZZaVOgPcxdng0Xi_JMKvlGYmcUvlTfZMm2FYQ7pTrF20xWUreNaRQqtU6rlWB3kkuPmMNBALyTBs,&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,DO_wUMKwP6osP-Lcj9Tsuc0xOfgXkt-AeVu4ULY02dHEgw5EY3nKNphpGPqFO3YW8Q8v2ha10FYjj1eZSEk8DnSnWAvwZqqe5UTDD-hIo-a3cjd2xwk,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,4iCAaiDF74c7cCKTdUIIlT6Qu6BmbQItK-gaAIrQtpNdie21ccXNS_dCydALepUtQNTgsVsVUs1NavHOugg0IlFK0CjTj8TKnThm55ffUeOx8zHc4oZUYpE_LRsz&typo=1&ancr_add=1
> >
> > )
> >
> > at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,SSiNb_ntMVKcpbykl8o6Zy5_dJ0Kox8JfbhtZ-GWRQBXEq1byRJrmjqbE-m8XsYk2KCZjykhzfVKFgcnYdQwoH4uNy9RUFLMCgnu9prRsocj2HRywrm-law,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,qBGyQwtcrkAjVKJZ9cZzaW1ivhGQgGfhpcJUS_ww-vfcQVNfD9OjDwDX1XjpzJWYpo08MqSOrKLpgO8d72TBVtSGg2OQ4AmggCv3IUgKeA,,&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,AA_FyBPbdHM_iFmpEg3W2LJcKL5jBsndJh7xPd7S51ckhU2gWbFmFseljX5EVAmxJQFSXrnashRfZnghyhaA3dL-AiiLHsxSKV8q_xsSJ18,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,Mm6fpsoFsFLUtanrkJ6cqlk2y1zCJbPDIfqxsd0pS_sfS05oTAHBNIXIUUd4iA8HJY0guikOejZmykugHqKeopxLvJE3UmpX9YX1Y3Gu2ToK-B8aRw,,&typo=1&ancr_add=1
> >
> > )
> >
> >
> >
> > 2024-01-10T19:52:34.435 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 54, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.465 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 98, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.474 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 20, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.483 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 128, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.495 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 158, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.499 [Thread-4]
> > AbstractProducerCreditsImpl.checkCredits:184 [] DEBUG - not asking for -1
> > credits on
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,PRaZ89srZpbRetuvy2A6q_H51waJMHKZwMIm19nHjdCbZxEVzh0MSm3wENcuBR3DqyjtAzHP_fuBYrRsG5jZl1sOJ6prXBGy7eOmaOc0pX9YvPQ8bDEN&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,BYjREAStKM-0e-e5OOY86CxL1g9WrAP01m5ROYtsGCMHTh3wox28vU8_r7Y9e7QwRiDtJ065Xst_zM3u0oFx_K7Fr-ZkN1GBxPR6aWEKrm4nQA,,&typo=1&ancr_add=1
> >
> >
> > 2024-01-10T19:52:34.499 [Thread-4]
> > AbstractProducerCreditsImpl.afterAcquired:98 [] DEBUG - AfterAcquire 333
> > credits on address
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,AVjYtSKN8IYY0tqPPLmNCShOm_eJRRCIMxLfSPfxXdm2DLLqF8eO5Kj5vXsOo8-8JnHebRNxN1JVuUNcmdVZr2uTToWlkSsgHzOe_A7Em9pERfdULzX2&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,r4o5lwz6oA1VbUpicHhTmEpC9U7BwWBMw8ZYMcrl4g-jVFykL22Cb2WJIZLOqv-hLbxRqgThYlmO40Jo84JsPHe-IYBjUU8WMQ-jy1lZqxq3YQk,&typo=1&ancr_add=1
> >,
> > pendingCredits=65203
> >
> > 2024-01-10T19:52:34.506 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 106, performing as a failover operation now and
> recreating
> > objects
> >
> >
> >
> > 2024-01-10T19:52:34.680 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 155, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.683 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 21, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.684 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 129, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.686 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 57, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.687 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 174, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.688 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 176, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.689 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 56, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.690 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 108, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.691 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 156, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 233, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 112, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.693 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 187, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.694 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 110, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.695 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 144, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.696 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 235, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.697 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 28, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 47, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 30, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.699 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 77, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.701 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 65, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.710 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 70, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 34, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 159, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.713 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 76, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.714 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 119, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.716 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 59, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.719 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 87, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.720 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 80, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 166, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 149, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.722 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 137, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.723 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 163, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.724 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 179, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.726 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 181, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 17, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 58, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.728 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 32, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 49, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 234, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.730 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 143, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.731 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 42, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 136, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 178, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.733 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 55, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.734 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 95, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.737 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 48, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 16, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 146, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.739 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 22, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.742 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 43, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.743 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 130, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.744 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 36, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.745 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 60, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.746 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 117, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.751 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 14, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.752 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 81, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.754 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 93, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 138, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 100, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.756 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 62, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.757 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 125, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.758 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 15, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 99, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 44, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.760 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 172, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.761 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 84, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 79, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 122, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.764 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 64, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.765 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 88, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.767 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 151, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.772 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 11, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.773 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 96, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.774 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 27, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.775 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 73, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.776 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 90, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.777 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 134, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.778 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 35, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.780 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 118, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 186, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 41, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.782 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 31, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 150, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 145, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.787 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 127, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.789 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 132, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 170, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 142, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.791 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 53, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.792 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 86, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.793 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 147, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 115, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 12, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.795 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 135, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.796 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 68, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.797 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 103, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.798 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 141, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 165, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 18, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.800 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 33, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.801 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 116, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.802 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 124, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 26, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 50, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.804 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 75, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 85, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 69, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 97, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 101, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.808 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 153, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.809 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 107, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.810 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 175, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.811 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 66, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.812 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 111, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 152, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 45, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.814 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 61, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.815 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 113, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.816 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 105, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 162, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 171, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.818 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 173, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 184, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > 6c945e66-a227-4e1e-999f-2ab339d9c812
> >
> > 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
> >
> > 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > 185a677f-f344-4973-9d3a-e01eeb102d30
> >
> > 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
> >
> > 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.cache.invalidation.topic
> >
> > 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.cache.invalidation.topic
> >
> > 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > c00b7a09-540c-4b59-a7af-99ef76d77e57
> >
> > 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
> >
> > 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > c3172442-182b-43fb-bb62-f47e9cdbd340
> >
> > 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > d6e3cc7f-ce14-4417-9453-6f29f1147b9b
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > c27986b0-f570-4599-b651-7153b4f0fa40
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address c27986b0-f570-4599-b651-7153b4f0fa40
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > acf62840-864a-4eb4-a902-4861db94eb93
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address acf62840-864a-4eb4-a902-4861db94eb93
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > b2d50731-8cd2-48a8-ac25-b541eae13f55
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.authentication
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.authentication
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > 20f25377-43c3-4f8c-a40d-c8400c63589c
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.permissions
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.permissions
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > fe52afa6-a998-4101-b382-495d8e48cd52
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address fe52afa6-a998-4101-b382-495d8e48cd52
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.version.migration
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.version.migration, needed = 65536, arriving =
> 0
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.version.migration
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.start.project.execute.2048.250
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.start.project.execute.2048.250, needed =
> 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.start.project.execute.2048.250
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.message.logs
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.message.logs
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.cache.invalidation.topic
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.cache.invalidation.topic
> >
> > 2024-01-10T19:52:34.820 [Thread-15 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.afterAcquired:98 [aoy5vnfnjetz,6ek4wlg8c0ad]
> > DEBUG - AfterAcquire 781 credits on address rpdm.v1.authentication,
> > pendingCredits=63974
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > 6c945e66-a227-4e1e-999f-2ab339d9c812
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > c00b7a09-540c-4b59-a7af-99ef76d77e57
> >
> > 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > 185a677f-f344-4973-9d3a-e01eeb102d30
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > c27986b0-f570-4599-b651-7153b4f0fa40
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address c27986b0-f570-4599-b651-7153b4f0fa40
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > d6e3cc7f-ce14-4417-9453-6f29f1147b9b
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.permissions
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.permissions
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.message.logs
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.message.logs
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > c3172442-182b-43fb-bb62-f47e9cdbd340
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.start.project.execute.2048.250
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.start.project.execute.2048.250, needed =
> 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.start.project.execute.2048.250
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > acf62840-864a-4eb4-a902-4861db94eb93
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address acf62840-864a-4eb4-a902-4861db94eb93
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > b2d50731-8cd2-48a8-ac25-b541eae13f55
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.user.group.query
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.user.group.query, needed = 65536, arriving = 0
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.user.group.query
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.authentication
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.authentication
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > 20f25377-43c3-4f8c-a40d-c8400c63589c
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > fe52afa6-a998-4101-b382-495d8e48cd52
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address fe52afa6-a998-4101-b382-495d8e48cd52
> >
> > 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > c00b7a09-540c-4b59-a7af-99ef76d77e57
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > c27986b0-f570-4599-b651-7153b4f0fa40
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address c27986b0-f570-4599-b651-7153b4f0fa40
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.cache.invalidation.topic
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.cache.invalidation.topic
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > 6c945e66-a227-4e1e-999f-2ab339d9c812
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > 185a677f-f344-4973-9d3a-e01eeb102d30
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > acf62840-864a-4eb4-a902-4861db94eb93
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address acf62840-864a-4eb4-a902-4861db94eb93
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > b2d50731-8cd2-48a8-ac25-b541eae13f55
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > c3172442-182b-43fb-bb62-f47e9cdbd340
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.authentication
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.authentication
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.version.migration
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.version.migration, needed = 65536, arriving =
> 0
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.version.migration
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.message.logs
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.message.logs
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.canvas.state.query
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.canvas.state.query, needed = 65536, arriving
> = 0
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.canvas.state.query
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > d6e3cc7f-ce14-4417-9453-6f29f1147b9b
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > rpdm.v1.permissions
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address rpdm.v1.permissions
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > 20f25377-43c3-4f8c-a40d-c8400c63589c
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> > fe52afa6-a998-4101-b382-495d8e48cd52
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> > credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> > arriving = 0
> >
> > 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> > credits on address fe52afa6-a998-4101-b382-495d8e48cd52
> >
> > 2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:184 [z7wx0g93sdr7] DEBUG - not
> > asking for -1 credits on acf62840-864a-4eb4-a902-4861db94eb93
> >
> > 2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.afterAcquired:98 [z7wx0g93sdr7] DEBUG -
> > AfterAcquire 505 credits on address acf62840-864a-4eb4-a902-4861db94eb93,
> > pendingCredits=65031
> >
> > 2024-01-10T19:52:34.824 [Thread-26 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> > captured on connectionID=d43c72b6, performing failover or reconnection
> now
> > ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> > Channel disconnected]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,FE9-P1M06_4-hpSI_YpzPbXBXnJJatopeQD2ZYxu3Pf9NWzlCotC1BAbSw-CkPVwd4p-ZF9OLD5UXIFMFY3bZIwkHhzIo32b1m0CpruDodE-_f-W&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,dWlUyTFJQxRIqOG8poa5pxge5mYVJ48dUXfACv6LUtKsZzFSVGKyZagOScZPGCnB6cxsyzdobSyk8B6Bwd79504jcLncGDwqMeY-9K6dpOs,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,IJ2rIXxWuL4HV_0QDasN4Pt8dv2RRaruLAddl_V22qHM09tvT787Et3f81LH8GEe2qDoNwQVHNYo108yuNu3b1QmP0RABo4i8D9b8fEGVe2HYd9QgjXy&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ZNE9b4nkKfo7M-_v-eXIZ86i5mBDuVpXhcuhH7d4Z3TXAYIWBA7iz2F7WJKgkGmWz9vMjcTKd3WNKgAqq_OrNuwfMP9rqwnZZdfDvOywCznI2b71&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,jYeMQW0WCHRb7cuJ1CqLy_AuRfdEcEvEJyhW319lWJzbpHyQUcNdPzWURfHTEtR71d2r_2lEhQf1hKps5IbaEzZsNJIWqMVdW1EGF781kPIabPU,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,ud4sg1bGnGSsxR72jGNWKmRRRqC9RhLjc5Fa_mAH3joB4qQCFRRYETYRnfp_427BSgD3-id4mG56z7G-3qi9hl8xOglhqXamz4f_kpydTW4Drw,,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,LKodAPH3Ka3eY-QiWDHN0Dy_mvuaRX03ueKTXS9etN2qHhzBavU5TvH8lfQjoDkin1OiPxB9M0-rKA8tOzft_WViz6TpZZ6NNkERWJmT8-UKzfsFuw,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,5ea91cHeAd9MIKFvy9sbzhVk4n_LaL3aA10YRDmCls04v12YLb2XMmXw6eBUcLVuIE_Z644AmEt_e0ljvQnrS_I6HavZOLh4zjfiHWUvesg,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,3EuUPQYl08ZdWQB64BkecRgHHYvSChwe8vbP1kub9GpHNfB5OCg1ZRGw767893o5_ljNDyszaWPa4bDjjEuzEB5SNMICoB0-cPac73Tj-y5RWpDaKHCkuwVo2L4,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,jDru8ObAZFnt6UJuRwWa5fbG9hi1-FGT2jjgjYcb0j9MnKP4NE8WTbKBHSHyKvKnv60VBWqMlES8saZuop8zduWr_U-bg4uKdFl24xth0Wc08-CvUBpd&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,buy5utXjIbe-DcJ7PlEjzXqjMt8mDipzcDzczhLR-izL0Nh1zK5MFk42iry_zn2VPAZ7CZ8GCTQjVt68Y73BXII1KZvBi1ATYfTqOscCWbX0DnlP-bE,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,FCoOt1eEzJLmEhVMnIIHAKa-kTFceMfQD6Kcwc4ZSxQI7u6ROHuAf16piW1PXfz5fwuSX1g87RD5aUJh-U4htAPPBkPWT1OqrJG8TvqS_LoMCTdPQPVU&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,uJwPcj3QaOxYK9Va_aF3DQjRZaOd6gkORxphEXx8crbnrWTQpbS4NUvdGwXFXV2QxdoFEiSVdKtLtallasFVmS_ThZCt_nFVtdxSiOhZB0BxRxxTXdy8ETkT0A,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,RozNWdaPDnLkxzHk0PIIu7yOI0mq7J36SJtiYqHJSj83AaF6otFCOMT6S0SrKP0i5_HdKBbJFOfyhgDON-XzQcdSOvC2Z4N90Agiv9ZXimQSY87tDA,,&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,cU6yMpGHxe1nbp7Kzrg6EeEWtSIupu9-5fpY1zduuLu7sLJpcOXhuMMDMhuVH92Zbgrs9uoRcfmFq4l74iTpclqRJEOpXUr_QjhNgLoE&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3xUOFzb6MCi6u1j2LRQL6CE5BWbu5eM6Ir7EsfF2oCaYUBtqnA1r3VDYEmgBiVBb7QypDxbOzY9knH1MLZZ3oikFpY_p69_553LHOTbV7g,,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,flQWOTe8aKezv5fxKav9FPsBjUaARCUkJOdOEGZrl7wCvt7GugyNdNvZClWXHAEd9rpj_ZWpZp3Cafpzk_fnpUQ4T8nZmy15zot9RNiXjba71GXeOkk2&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,9xoBaA3kGi2K8zQ8BuNLDq0dYmsnE1nuJw9sP_cDKKpUiUl1eHBvRBJF127g2yN9LNqLqZkyPVSq8YusAWP6k6MCI4_4n846oL1LY84Tow2q00oOukNL4Q,,&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,i6pIMkwS1l8fvBdFueQgC2fd0K8urWRRwGCKI6oTBX3N5vAYCxnNUJBz41YEp0I3NQ4S4ZHQ194n_z9O4VXiQ_Jz-cSzO5g2wCy4Jjpoams,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,OqfGI8b5WUrx4fzs6Vhc7lLcb55MI-F2H4j9ic-9Fbs1_FhBfTLzdobmjbVxUHuNV9kcphYOQdlHoTf8bN7hu4yIeKsdhwLW0rasAMuEm4mWhNV8t2c,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,j1hybmOnVH0IFEuunwALaXX1VhMjfTNyNAVYi40bd7VIdfk_jOox9LkDhVy_uodaH4IoC4x4qyX6rCNeuIoEnqF4PjDggaSOdEh6HJe1idmx4NGO8AI,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,ml--e7XB5_TgtwWFh6lw23pXHYNiKpjFGKoUKmwrOwHyS3lnFQ78O6WUjq-OM1ITxXTrgD6P5K2E89TnqOgGcaOyn6aLaVIcvPhARMyQEqlFDI2mcnYwIfwADKB-&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,QIlD68RsucjML4is9S32DJRE9Z3dapep-RlKt-pyrEuRTIt_G8nWBtRgb6Q8IMi4IJrlS-LAnOypotHiXNa3NnJWi4t7uDjt4su7nlkZ_w,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,olHwwJjC2TUJmGxt3e1EpgRMfVMAAdmudKrqZ5ZByilJc51VAK_ZZHGL1mLqUqv59mtQqrIVOT0FHxcuVGnfmTsHrtP_Lfh6vEjgZmLr0Aa8EqAKSU0BDQ,,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,oMqaoHGrFSMkCOPl4uTKGzTSGTpfT8r39rhdKWVa_mFBDphxOPM_6yaRZJfLYNMa0AyN32-362kp_uiD-mRcEgDlEH0vkXHwcnpH_v5piaep5XwKlX3fJLhn&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,3crdMkhGhlkrJXp-HsjZIEJOjIoajfvvI_g-uTOHgEc26-rm5spEwaDR4tTrIOKeN5HSQz3HfKcnanjUMfuquaMfvk7M7K2AtlUVxZdR9Q,,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,3aPmcLDsyuHhnVLEv2uuGHy27dRF0jG1E2N7oB2o3YLd64XVZE-Wu8S5GjeDGqfzFEPFWOKwMR990Sz5Rl_09u_nU65WDNi2AS9_hDvMNMsxtfkYZuZNMQ,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,Qv7RvLuSjcsV2_72qHHLM6lmOI05RppEGGyR8280SVuKWQgOAl-EeFo8ZDeB_9Hw2f5ZgHbKDJbuB11rT-FBX6wf71ASJsIGNZYXzHn1HnNPy3pXaBaEgvrJvA,,&typo=1&ancr_add=1
> >
> > )
> >
> > at java.base/java.util.concurrent.ThreadPoolExecutor$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,KLgaXgh4AHKpDe86W-r95VdTmG5KxMAi8sJty_SCxIHdKsN17C5SgWizyFoSLnNCx82oAkl1IQ7G3WdltO092d1YUzVphf699iq64LfNMcdXCeZV&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,7hPEKeOrrW0p5qUGMytIu7avFNFhxNhXFzKgo6a-hDqm4ky01dphgxAxTpF5DmP3oUrVV1uWYwMbJuUFv1IoX_o9-H7Y6xXWdj067c1A&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,6t2DJJUOLlA8obI6HZE4TmU6f_q6BKzPgJBJoo3fUc8j9wMievz5OvhrLMW74jh0Dtb_PEqVIlw_zez9B1EfFhl9WokvVE9IvfQ9vscZDtW4&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,dU7lk32qJcXeW-cUtM1E65BB47JiLhSgJYTqhLIX373Pj0miueYi3kN7MYtAU5zT39qOtzzAiHcBVYWsGDB3N8R31seI_xbm-rQzfd0jzb6V1rSLy1kotiI,&typo=1&ancr_add=1
> >
> > )
> >
> > at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,lKCCRLBtMK05PYln4EyoJ9wkUFDWR-bNCq-w2CS3bTb9B4_6-dwwglQPojSGYr1CeGxZdR-SeX4AkZ9wUq0gK-sVV1TN2CgVQFfAev2-3EQ,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,BtR0iFYtXaxmAlygBDvC74BLAVELzJivsFTs_YS41zS_XagPPAr0UUKW1lNrafWE3WxaZcW_S5WbNn583_uHQ7aLAA9vskUJfxaMwUrM_DxAOd2N&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,JM5uHqjx69DeHPuH1fzlCWUlAJt1HxJ7SMAoo16B-QlHBBzRndsmUvfqA_ZY1S6EPFV3mEGZBPAWdJteWlF9vXQYKEvGK1kyswVvG_maoCFPXHC3niDtcVnH4g,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,nW_o_ayhGlEOytZThvhNsFjqKDq7oEaYAEjw_jVKEHAnXglYBlSGSA3yj7wyG-2DXRXiRZSD11UwsKx7fSGPuHkUUSoULIYqacVjSrfmUuEfG_A,&typo=1&ancr_add=1
> >
> > )
> >
> >
> >
> > 2024-01-10T19:52:34.824 [Thread-19]
> JmsStaticConnectionPool.onException:78
> > [] ERROR - Receive error occurred. javax.jms.JMSException:
> > ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015:
> The
> > connection was disconnected because of server shutdown]
> >
> > at
> >
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a714&c=E,1,J-35FPJpVlAlzxdK4DRXRFHe8NLDGoFyrF5bd0TJMALAxu6IPXf7R38xF3V3S_obALUyJZio7n8nSS11ImujLD9S8AoKJWyIuS1mhUUsFA,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a714&c=E,1,3FhMyZ8H2FihvFzilmw5MqmH_tkqp5vHbCOJsN5x6sQNSLyFELuJt_Ej2M_24CH42Xx0iQesALGDJK8HHdR_PDaLploHa8R84-uwfftA3DUM&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a735&c=E,1,6plQ94hoZFwDxCVEMlguzKCU53o7QqHp52VpGSjXXTWicH1j4n6pp_oAxzWcmYFeQfPGabu5xKNPXO5t1rOzVCkEJL02B0Z0C4VpE1nN_Q-s&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a735&c=E,1,JvpZWdXDc673FDsIBWz8WgGu29y6j8H0zAczvf_UXYyUAzEMKtvXJ3tMkleaFsrHU-QdlAUi0bWddjnikuurWEEMfZs0jIDWZ3jnrx2__Vbg93yZ8oo,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a868&c=E,1,rouSt1axPk8dJSJf8RicvrYa-2QJv7ZdOGqbW06Dh2gGQv3VpFGKm4kveVLGLqGB3MY01qSV3dPGxTpcGGuAtlDEdii8a2mZYz-33WOWXnSkdPrCQbjy&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a868&c=E,1,AznnL1cSXDTbCj6iQ4ovEeP71-oopbCcH0laxV9dB-Y_kBuerNfeOAXRkRLybAqgqQBjJZKBNFEoveCJjTGncMCguHnY0hvsdZuKIHz6BEhvRX7N&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a856&c=E,1,53nm3gPRPTzcTWl5-TjBB6-0L9iUvYzlHKumiGfEb-1fbfZStPd4a3L0h41Kq1typVYdgSUQ7nT3w2U1ZuhBKEa3-WhSc85ptfbZ7I3VciWzT0Yaggw,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a856&c=E,1,vBewrTh4rfp_ezwl572jqeiy0VBShHwQJNRT99yv4B54VM5Qaf2hTHrOq33emXwk_WSFrcueJSCKI90AxcN4jUznfP1ZLzL1dJdUz01zK4vN&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a802&c=E,1,G_tte968_c7aOAQgyS3Ugbr0nl4R33yoPsC_W4flWhC16Em8YPUdOwKDzassaWqeuYxIOh5QXYq1l7ytew3cb9sGIgX_of0G7wkTgZJolWu0wekM1g,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a802&c=E,1,EY-ry2DzdJloXWtaemW-dWnRy1-rr2mQFFjoCbmWjX6xHCNZF95ldY2zUgfhOhSQFfzm7eYV0PdRUTOwJrx9jCD4U-B5ztFqWmvqEmJQuqdCyQgypA,,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,o6cWiT7bpmgtzTzbYb3pAQFFxJ9bN5oZXHVo2EI_ad4rkkLBPQcK2Q7N1UbZ0ZmqCgG3shVKqOI8vF8UYSAVxzHE8bN4FoWJzNCFQuGHw0X11iUlFHdX&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,gHouFFMo7plejV1zi4ureD9QEDkW-WjvytRlH6DOyEt7oMO8ULkj6APov-gDhxtXAspTlk7qF6RYw-RrCqfSrXlcE0Cm45J61qJHt5Yc3zuBJJyEX8A,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,EYjW0HIGARwXmmVt6jB4dFiC-InWYcDZxydanyFK-waOk_olLAwd-yudL9xJHMzPqJMuKQqjns2IvmTemRxjKodxxofm_Zv4DqvDa5X5OxQzr-_L3Pz113KtNag,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,17MGSyeFWTwZKhb2DCqWeyFvdlinlKAunc4JSHJltjjiaCeT_1UjXFsTMtEiF_NYIOMI3brJbPkRW1QkJIhe_Br44jy1od8-BI3l37IfFJzZbcculrc,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,d4k7cBoCYIC9j51ETZxBDEayFc1056FwCQHT3d9loHqh-7uyMyVqmQtyevBar-ule7SfuAHPBx-chAkT7YfTODXaobYfOO_9b6if9LIg&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,F6NyfiAiJI9fxHnOQnnpO7dLyoFavqwnNivvLL6x2LO54G8SpNhXjlUN748SkNWwe_oCD9JhI7u_BOk1uNbwSTYV0lXRyewBrwvvysSEOSCggiFCFRgjtA,,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,Esh9CwiY8Q8wRaXAM1ofCeHlDikAoWnYBv0KxS4ZX7xLt0h0R1rYpHtiLaELOsdwxhkfV9L6N7QEWOc4k-RiY6aOzby7BtUd83a0wTnmzJr5qrfSgGVEaF8,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,TnFdrKl6nCZv9PR6lSsRZ2YN0r8LqpGQMRo3oHlgyE7_AgUMzrW12R4HeB5JhX8Akv-1iKeVGrnTLV2WeAbBOCM-fukhfRPsdUgq5pwe0yU,&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,DVdY4t00amwsXk2vhaH6zLZaOeXBORNxqLIn2J7uyd6muQgSoF-SBrv0E5EGGpc4YB4mgXirpL7_C_uKsTlrYDo5DQH6IceC8gUL4vT5jErZTUFdrTHi&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,vAtPtyOuzaufOELKjMWnOBMrJEOShH0EzH9G3WL8gak4hWHmnVWPL1W7c_YnxRH2IerBQErg9VJApOiiX6TR6tcPau3xqRHrPyrsH70KHg2v&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,1QUkZh--OnzafOeQWcKcE3LYcxZBZTS7sFTwbWk984gjzF9pqi5lqW4RgsCfU_g1nLbrmd8LT8IEYDfHskjKl8sphe40gTARfNmI4gGx7w,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,pak9EX0xrMkRTN71voUljZIOmUvksqdCCqWdwh8fq2R4OZXUQFIHNH3P9j0t-wpQStfSs_ZsT0a9BZYopaoyMSnMkN3xqvXn8A9dTehv&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,geFSAtYIGv9JWgw2y9daWnn5EtrIqvUZaryRk8-l91NF5pD-L8bV4mxozd_mcICnm18YfXjygphpodR-DltMQ_vHwUd-4OWay36gD2Pwodoxtjs,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,acJEflBFObRkFNBskLYk7AARfS1_CzZH3yFGYB9r_Ype7n0DAWduZhGpmImVc9k1pTnmFoA8L8UukMVvWBinxujU6r75ipmzh4I_cNQ5Gzmd97g5SAU4dRsf3Hw,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,nUBUrDCvls3gKM5PlBD2cIXpf1W6rFXWSguqSV0S4dzfPj90PFVFAHWxCw4eO63hLAoTxi6uy8_2NIh0YHuWiBGYKicU_56rG8KAqSNsAWLRLDtAO0RxLw,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,aM5tVNMvOtLUxuoV8Dez1Y7fAS8KwME3peX3IqP_QPViIWgkZQahJW_1BMBQUo35u5h9hGJ3XSyDgkECwWOGDn6a9W-gctjh8WvUKx9yQ0sSLwheMcs5mhR2yVM,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,6p9TNE7Xs6QBgPslw7yBqXl57NwtLJGLN7k-pYgJu9zGoa-hmO07zvWfD_kO4CbadX3p99V2Y0lg94ChyckIxDQQqhJE1qJYz-J0cdZYYRpPQg,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,fUWcgLX3hupjGCjPYEByZ8xfzFx4iN-VnkLB9MarNu6OEOfrwnCkvJvtu6e-ZV-FREp0Ssisi_fezDu18eP9adS4mmcPPlJH69vnf0YDVg,,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,uWfs3OljpnBLADqOUnftuzzQ12hzV-_CIwYB7OuBK9OgZgUZTvMdZpSPOqxe2vDLn7uXOWQCAeQAz5kh4iSGoPSrMeWO8GoHNI52KZEa&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,KuAjbB9H9ddyi6unfXrVdNXylpB7l6qLIUfJARScyUnc7iH1GtvO0s1MAtB9fwkdqXq8idOJYZs_rg0iU-sN4aV24ID0Os5tsyvB34ba3ve2D3diROqixzkQuw4,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,uCnHZgtUpZrhEpoalGTGPF9_Zo_vFtwZpK7i9OECfoEAgHQAk9HXrc4Ai9_T2uwdRSTm6kuInb_HD57GvWWLNrX3hqs8ERIp5QXo50ME2UPVvaYugac,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,EcYd8WT9GtMN9IXo58gsQpj5uNwcdCRtDfa0zG9t2jpvfdaSX5r9vLey2icJ-4wo3EyBMyUQzwyqYwC5BMfe4yHGjyo7h8jxmjcEKig4V0KyVa_SVq58NBT8&typo=1&ancr_add=1
> >
> > )
> >
> > at java.base/java.util.concurrent.ThreadPoolExecutor$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,7xWJeT8e9at1yRzTwb5eaQ-EDFP7kbguZ1HlTyS15fjDzDNz2TZlYDBhylM75L9Y36HGRriKW9F69ImAAFS3e5X_53PDBRJOQPtATaWXMJlwz089&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,_a7OW7NorEoCm9eOnYjCi6TFl2__or_ickK-DwP4rrMmRX2Rj_TKaxpGjSAwCJZasllt1gefhShoob68MthMyp6bxv9N5fucG2HiF0g3yK4kqHTZjiFn7kxZ&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,_6j-wwzNJpJ6yfdA5D-GsS-QP4F7fbTyvCvRsTCwS_mB2qDx-lBWvBexxnGAMZnGwmMBeMncW4HkEczGZS6veAsx03mLcMheAU29hvOHDpRCJCAn6uwXUweChw,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,-ttWHQzQDZ94E-HqKtYTKV2nuApkTk60Zy25iWFIykqfNe0o2seEul4UE_gETZr08B40bJjmZKMHq_-R0M8bQ8Fbu8twSkWx9MPLFUhc&typo=1&ancr_add=1
> >
> > )
> >
> > at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,oHfUtnglQcVzm7c5ytqIWCqg3Ql1T66OiTaXoYJycG7aHXcRVA-XGunwLIzdVb0E3sYbVFE9yOTMe7CM9jDlIGgGtbkvhp2_FUk6hv74Kg,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,624T4PuCyLmsS-UG0IoXRx3MMz4DqnU_1BuWfOz973WSpqfLsFFDAICu75KCoCwAs9O8WgKTEcBVqRhhf1JUqI3Znxem3LsG7Ml0Td3DnJJUoGYR0BE,&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,nsf2ybbdjymP64mZ0JsCbzYoQjO_4UlQcKyh5v3NAla2uaTeh5sLx0V7CeR8YWgzcfu2SaTVEzv8aGCf5ha-xcrCH77k0UAnZgPtmcXMFImJxbtxC8ZJD8J55fHI&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7UL50c_ahB0IhSHbe9OlGABBKKE4OFyNM67SjmNrSo_LQhfViHAlLgvwdlRpbmDdymI5VaeOEklHjvlweLBz8Ske6wjc8J3CzM05VYenh6DzNiI,&typo=1&ancr_add=1
> >
> > )
> >
> > Caused by: ActiveMQDisconnectedException[errorType=DISCONNECTED
> > message=AMQ219015: The connection was disconnected because of server
> > shutdown]
> >
> > ... 7 more
> >
> >
> >
> > 2024-01-10T19:52:34.828 [Thread-17 (ActiveMQ-client-global-threads)]
> > ClientSessionImpl.acknowledge:1225 [] DEBUG - client ack messageID =
> > 10054445
> >
> >
> >
> > 2024-01-10T19:52:34.862 [Thread-24 (ActiveMQ-client-global-threads)]
> > JmsServerBase.logRequest:117 [] INFO - REQUEST IN:
> >
> >
> >
> >
> >
> >
> >
> > [image: rg]
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,01YVdf0IHIl4w2xIwUh5i7ToRiQfv6TxT-FRWSNW0n6e2aucHkWVyKxkYJTpG1jt7_EUCuQVS0NWGalDyf11AUIb3qVPj67IUtjJB484IdLKrHgbmSd-Q8Vg7AKZ&typo=1
> >
> >
> > *Lino Pereira *
> >
> > *C++ Developer, Redpoint Global Inc. *
> >
> > 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
> >
> > lino.pereira@redpointglobal.com
> >
> > *From:* Justin Bertram <jb...@apache.org>
> > *Sent:* Tuesday, January 9, 2024 1:57 PM
> > *To:* users@activemq.apache.org
> > *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> > advice (continued)
> >
> >
> >
> > **** [Caution] This email is from an external source. Please use caution
> > responding, opening attachments or clicking embedded links. ****
> >
> >
> >
> > Both the "replicated-failback-static" and "replicated-failback" examples
> > run without issue out of the box on my machine so I'm not sure what's
> going
> > on with that. One thing that's odd is that out-of-the-box the line 122
> [1]
> > doesn't match the method call in your NPE. Perhaps you modified the
> example?
> >
> >
> >
> > Aside from that there were a handful of questions in my previous message
> > you didn't address. I think the questions about thread dumps are the most
> > pressing given that you indicated you're deleting the pod. My guess is
> that
> > the client is getting hung in a network call that doesn't want to return
> > due to the pod disappearing. A series of thread dumps during the minute
> > delay would confirm or deny that.
> >
> >
> >
> >
> >
> > Justin
> >
> >
> >
> > [1]
> >
> https://github.com/apache/activemq-artemis/blob/2.31.2/examples/features/ha/replicated-failback-static/src/main/java/org/apache/activemq/artemis/jms/example/ReplicatedFailbackStaticExample.java#L122
> >
> >
> >
> > On Tue, Jan 9, 2024 at 12:47 PM Lino Pereira <
> > lino.pereira@redpointglobal.com.invalid> wrote:
> >
> > Hey Justing, thanks for the help.
> >
> >
> >
> > We are using version 2.31.2, which is pretty recent, if not the latest.
> >
> >
> >
> > With regards to the log traces, other AMQ threads show similar behavior,
> > and our system isn’t doing much until AMQ-client stabilizes.
> >
> > I am running on a k3d cluster and the failover is triggered by me
> deleting
> > the dm-activemq-live pod, which k8s then spins back up.
> >
> > Also, our slave config does NOT specify “allow-failback=true”, so the
> spin
> > up of the new dm-activemq-live instance does not trigger a fail-back,
> which
> > is intentional to eliminate failover-failback chaos in such a short time.
> >
> >
> >
> > I am trying to setup a simpler test case by using one of the examples,
> > artemis/2.31.2/examples/features/ha/replicated-failback-static, however,
> I
> > am getting this NPE on line 112:
> >
> > Caused by: java.lang.NullPointerException: Cannot invoke
> > "javax.jms.TextMessage.getText()" because "message0" is null
> >
> >
> >
> > The NPE seems to indicate that, you cannot call consumer.receive() again
> > even if NOT acknowledged, which seems to contradict the comments on the
> > code. I have run into this same behavior in another example as well,
> > artemis/2.31.2/examples/features/ha/replicated-failback.
> >
> >
> >
> > Can you please confirm the expected behavior of the example code in the
> > 2.31.2 version? I can then modify the code accordingly. Or please point
> me
> > to a working sample code.
> >
> >
> >
> > Thanks,
> >
> > Lino
> >
> >
> >
> >
> >
> >
> >
> > [image: rg]
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,lIOPl76F2R4EtenIAEJMeQ17kfmHQ6buC7G1W5TRefOcSfxvj_-3dLXV3MKDkCTkac_65Rw0SYmof66i91zGwgb_SS1nKaBpao_Lw7JNNk1mkE33&typo=1
> >
> >
> > *Lino Pereira *
> >
> > *C++ Developer, Redpoint Global Inc. *
> >
> > 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
> >
> > lino.pereira@redpointglobal.com
> >
> > *From:* Justin Bertram <jb...@apache.org>
> > *Sent:* Monday, January 8, 2024 9:20 PM
> > *To:* users@activemq.apache.org
> > *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> > advice (continued)
> >
> >
> >
> > **** [Caution] This email is from an external source. Please use caution
> > responding, opening attachments or clicking embedded links. ****
> >
> >
> >
> > > Now, the backup process does start processing msgs right away...
> >
> >
> >
> > What evidence do you have of that?
> >
> >
> >
> > > ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute
> > apart, before it stabilizes after the failover and is once again usable.
> >
> >
> >
> > The log snippets you provided show a reconnection attempt back to
> > dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to
> > dm-activemq-backup-svc:61617 both of which fail almost immediately. Then
> > right after that another connection attempt is made to
> > dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e.
> > from 20:03:44.874 to 20:04:44.933). Are there any other logs during this
> > minute? Did you happen to get any thread dumps during this time? If so,
> > what did they show? If not, could you? What was happening with
> > dm-activemq-live-svc during this time? Perhaps there was something weird
> > going on with the network related to whatever you were doing with that
> > machine to trigger failover.
> >
> >
> >
> > You might consider simplifying your test-case and then adding complexity
> > bit by bit to see what triggers the difference in behavior. For example,
> > run your test with the client and both brokers on the *same machine*. If
> > that works then move just the client to a remote machine. If that works
> > then test the client just reconnecting to the primary broker upon restart
> > (i.e. no failover). Then finally put all three entities on separate
> > machines and test failover. Hopefully such an exercise will give you a
> > better understanding of where the problem might lie.
> >
> >
> >
> > Lastly, please indicate what version of ActiveMQ Artemis you're using.
> >
> >
> >
> >
> >
> > Justin
> >
> >
> >
> > On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira <
> > lino.pereira@redpointglobal.com.invalid> wrote:
> >
> > Hi Justin,
> >
> > I’m working with John Lilley on getting an ActiveMQ HA configuration
> > operational. Our current HA config is replication with master and slave
> > instances and with static connections, and it seems to be working, except
> > that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect
> > itself after a failover from the live to the backup instance. Now, the
> > backup process does start processing msgs right away, but the
> > ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart,
> > before it stabilizes after the failover and is once again usable. I have
> > included the relevant log traces from the Thread-0 ActiveMQ-client
> showing
> > the Connection failure detection all the way through when the client is
> > usable again, over 1 minute later. I am hoping that, based on the trace,
> > you can find something that, perhaps, we should be doing differently in
> the
> > client side?
> >
> >
> >
> > Here is our Client connection url:
> >
> > (tcp://dm-activemq-live-svc:
> >
> 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
> >
> >
> >
> > Should we also be including the corresponding connector names?
> >
> >
> >
> > In an earlier email with John Lilley you recommended 2 brokers using
> > “shared storage” instead of “replication”, which I will try next,
> however,
> > since this issue seems to be on the client-side, I want to make sure our
> > client-side is setup correctly.
> >
> >
> >
> > Thanks in advance for your help.
> >
> > Lino
> >
> >
> >
> > Thread-0 log traces:
> >
> > 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> > client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> > failure to dm-activemq-live-svc/10.43.197.248:61616 has been detected:
> > AMQ219015: The connection was disconnected because of server shutdown
> > [code=DISCONNECTED]
> >
> > 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> > captured on connectionID=a8a4a77c, performing failover or reconnection
> now
> > ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015:
> The
> > connection was disconnected because of server shutdown]
> >
> > ---
> >
> > 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> > reconnection attempt 0/200
> >
> > 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> > and currentConnectorConfig: TransportConfiguration(name=null,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
> >
> > 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.197.248:61616
> >
> >
> >
> > 2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> > and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61617&host=dm-activemq-backup-svc
> >
> > 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-backup-svc:61617
> >
> > 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-backup-svc/10.43.11.76:61617
> >
> >
> >
> > 2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> > connection been made, returning null
> >
> > ---
> >
> > 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> > reconnection attempt 1/200
> >
> > 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> > and currentConnectorConfig: TransportConfiguration(name=null,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> >
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
> >
> > 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-live-svc:61616
> >
> > 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-live-svc/10.43.197.248:61616
> >
> > 2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)]
> > NettyConnector.initChannel:762 [] DEBUG - Added
> > ActiveMQClientChannelHandler to Channel with id = ced14fbc
> >
> >
> >
> > 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG -
> Connector
> > towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> > httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > failed
> >
> > 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> > to connect with
> >
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> > and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61617&host=dm-activemq-backup-svc
> >
> > 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> > [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> > httpUpgradeEnabled=false, useServlet=false,
> > servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> > using nio
> >
> > 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> > NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> > 4.1.101.Final to dm-activemq-backup-svc:61617
> >
> > 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> > NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> > dm-activemq-backup-svc/10.43.11.76:61617
> >
> >
> >
> > 2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> > Connected with the
> > backupConnectorConfig=TransportConfiguration(name=netty-connector,
> >
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> > ?port=61617&host=dm-activemq-backup-svc
> >
> >
> >
> > 2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG -
> Reconnection
> > successful
> >
> > 2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 11, performing as a failover operation now and
> recreating
> > objects
> >
> > 2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)]
> > ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> > reattach session 10, performing as a failover operation now and
> recreating
> > objects
> >
> >
> >
> > 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,vvTQQyz_c1orZi3hJuq2ry1TKwMxLwK-5lFF_d5xs59Uqkggbs33ePLHKsLtFDFcSC0poz3LyKicF4cLJqyq7vAKuqJFApvRj_dDFYycgco,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,hqiYs6vnJzfq-X42G9W9ohJtYgn9LRPR-82RHxMyPnKUD5xBwCbo8W6b-PegfmAGwvEENlDO0k2IkoGFFegGxR-H6J72UHHlpEFbGtJzhCF3sucQ&typo=1&ancr_add=1
> >
> >
> > 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536
> > credits on address
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,W47wSuoEjpx7WYHr-JzlVVX-Ynfz-605gpt-36aXWsY6a829my5okqJZIqnhM9xZ55Qg3K1p8v_plEA8dwM51sR9xD7gamJmHMziyTUbhngPiHtc&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,KX9atZvl9XxpGPWobW9hXsmaLrVekfw2221zvD0PJ9Ik3fEQVyHKcZZCe7MQRciamxD-T-WDEk5Xa7_cAt0PSHwVW3jG7zzhq8XH9XIRC8EXmg,,&typo=1&ancr_add=1
> >,
> > needed = 65536, arriving = 0
> >
> > 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> > AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536
> > credits on address
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,BWFGc-lOAMobDFV3VeJDY2Kuijq3pFDr_VCikUkusDdeYbb0Vb0sQyfeLak8kHOd64VVSCsk-i8q-_Ow4H077PR_JbUcjLwm7c2eQxbcvpB7_i8OP58ULr4,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,ZOCwWoQv34cYLuw9hZvJzBHfSrX6oQeQ_3-SetFMJa1hRC6lXgiKsAElOhfZ_vfnBhRGRJPZjSOqHIbwWO2ltSYYY7ElQjyItp0uDo79_jaDkw,,&typo=1&ancr_add=1
> >
> >
> > 2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)]
> > ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> > captured on connectionID=a8a4a77c, performing failover or reconnection
> now
> > ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> > Channel disconnected]
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,Sofx2L-Lek25HDuPLaBmuRbxNTjT15ZAkPaa3bt8KdsN8nfIAnfK_-i_LGVt7VIPBJAfTKCSvKdhtqF5fwMcZeiqsq_iCMdD1bBW62tm21E,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,obbNTaFqn6mglgyZXy1S-0nBLNVtEYlkZBwAEKSGDGN0RuqlR_-7fIQPW4ELNqVFJdenAcKh65C3yQ2b_Lef7W01Tst6ZTrmI66KfHLae572jXv-HF3O&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,AdTG_wVqCegqwIBWNNy815gGCWjYHluAdoqWg___kmDOihDqWodf-WPqu_G60ChwqSuOGHLMOmsRc1Gx1sxmV2odmtdcyxMRdFwNaw7sC18VSdo,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ytVVTXzfq-J1NXl2ct7T_47F5EbfZNQTXP8dlqw720EiCe32c6CLwlpk0-v4MzUD61FErq0nkRTalFpcTtIN8TDeBvz70rzb__TyeIDm48ZKfFU28qQ,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,pMAiGcinEIywwqtI-_h5UmfhiVIsj_wZxAwWzZo6qzUyUe39gOo5cuTJkTLvLOnz5erj7M-o-QXlxwx3N0QsIOrwhB8B_GcuBeAG_CvIIxs,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,KQYUuEgOaR6V8a_ZC_xBpKZmDneB4LTL1cQzn2Gqtla8ntHJ1fgRs9KelLgvbMd9zvBTg8maXUerYZYQZa-DlfK7fon0JZ0O74PNJ1SOCHqIPN4fZQ6Xv731OGOD&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,5BrDFP93zUeig38-IRGrnzjTp7JfWTb1WzEc72yNCIoRDXcwnJ_1esXJshYjOciZ6vjMykJkBZ_6HqsaGLoyPvPLmtDnSuNDgOa6TcAJ&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,OSVh6NSORxz7qZ6eIuUDgWskoYfyXT4NNdjJYGMbH7HeWEUiymys2DnptjYKdmQfqjfHmI-egRZ31yWwd9MKzEd-gbbsuTUsqn_46niKaw,,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,McAPJp1t6gnApibCFe32C-8Cr9g_7ABZfKNnngO8C7OfjTCwbmlQoI-MGveBmExlr2kJuXuk3e7YciVvm-O_CJ_VqB6W638bjVcnc0rTsdoZ1qV6-i09SBemJYQ,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,DM2boEBGg44zzcd0m37ITkOnfDrkqLE8ZSFGJpEI4AGo2GyOI2yKX08AEkKHxw_UnDMF1vccYEvESFDXjXBBjjIGO5TEsMLWZHbJF3AOIEQ73uc,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,_8_xgPRaoPIbRD0yVdFOK2vlKMKwV-y_AyZkl51NnG9Bken-pX8fV_6ZnBLL_1M2NNT0EM1uFQ9SMYHiFv6Z7wYtXfzmARfem638au-oY_Tu7_4,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,wipgGeA7UVZIy1uuqvuxJEx4BA8TvYfSYmO617IgrvnOo-z7sn1vUjXFqIEU9M1HmedBVoHxB_LpvuEZ9p_biioPM8V-V9eS8i8oST46GvBhZZIc&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,m1mdePfZWYA-kcbFesMq8mnF6E-jEN-12F4t_ogHcJ4G_XcN6Dego1WHs9hZUnGwqRP0K9OW0EyJdfaKVUHimTHN_v3oiAjtz_1m4D92uUWM6XqiTq1AuMc,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,0FNTjscT_d2CNIid0J8d_x7C7Peb6_neE-a_MZeKqkM1o_sPGujvpkXuiLv6kms9wSFyYqSux86C1MX2R8WKM8IZBsjKt7sCs03U2lRYlVbw2eTDg0ihSEV1nA,,&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,_rouNwhPTTi2XkDcyj5UbDpjM5Xb5S4o5EbjNaUZZ5kVwkN5fUrTM6G9u83Q8t5MY2kKWp6eDBHyQ2XboNiD_3iYIZEN3rbdknbKtd2zGdiq&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3yKPqxpIi9n43QJSCQiysKGlEYttZ5Dmuirc7jUAWNz6orLfIUCYwdDEJ3eU0QiHywsNAJdWqrvnx5rZK6-8TTJb6o8tr1WSvSL7omLF&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,5f8NdiJ-aPnMigLbJtzN83B8k94qmJQq43nhmbgzr_xv-_XJ5Nf2ZDVZok6RUKOHjdb0vuJnzm-n64Eq2fnio3oG5I_tuNWTRbUXKAhU&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,6B4qLrj7q9DzDb0Trp9xgst7cOBhbD4hcEcGyPFul5E_mh3_oGmColB9YQkQTB1gJ6x5Mcm0VwvBLddpDpfFkcZwlzb4YNR9Ym5fcDclFW97pIrQ9oG4&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KYDC8kncdxnp7pATIxAxUBktC_iMOSb28z6Hw5CXdbb1busOZDrmEwOd0gz90nh_YCO_IYopJ2FTRxz4j7vr_0mRHXdlarXEc55kzAG2IUs,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KqR93A_gEw1FpU7__PsYjSkOC5LH0zFTSKu5K9YlCdVAz3h_c9EkBndO7aliE4RSbK_qslEZp0b-YzJfCOxmB22jAfplsYxguMl1qjC0rWpNjfr9_fh4RjNL3AOK&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,tiV6Vx6I0Xik4mg-tDUPMR48Vm95Asg25jpuyGMrlkTJBhdRMLVSEkT2KzpQ8s-Lxn5qQVtoTTc9EvJ1VH_fdwyMxShZj95-nohpqY0pNeoE7yE,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,mhChlKkihYjDw_Kk5SL1d_4eb7zhBlSzri_nwjOUsrs6MAvEfDAAjia07yTMUU6CQqc0WbDS2Fn8HJJI0UwJh1N7SKiz9dsP1gJLmgY66huLhXZE9c-3ECGXyIWQ&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,MtqIZFsxgj-Lx2vRvGSw7UH_Pzh2O5B_BYXHM7C3dLm4POkmZ79K15Xhs-ektW80o8GPKQrwPAdQYjjf_jDZ5b5nKqbmVeMU-qBxMZOF1Q5zPvBbgZNlZlU,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,nn8IDmg6XQJLLXzH7kunXDNZa_3A7fUlnHdRCLoha7gQWJO1QeY32duGFAQMYaFHsfIbHbv142mei8HfCqRwrpuUfikYxLkFoXaD3pORZb9roBJs0H8,&typo=1&ancr_add=1
> >
> > )
> >
> > at
> >
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,01Ld9YJlMjKksCSI3GimEsT9Tmtan8dI9-JcdLHZDdPqMABb2KMaE-oCZqJaiN9ZFK5rw0M8w2_bT7wc47JrPyfbbthma2evZrLIzSukQMFLpguO&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,DPLEvt-MPmmzv78LEkeXtYHM2q37m12TxX-nqjuIJurQdeXydsSakco9t_O7jgBbQ6y115xxdDb8IonFZbM86zgRDLcvUmeOSJuDmZybD_6JaTX0zpbQ&typo=1&ancr_add=1
> >
> > )
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,50rc-e6y2y_kWxF_bd6gVYJO_verct8DLqZaXGoNGN1m0VCTQyWRCDwOfScwBpKthaOqrPx6sg609tqQrG5vOR8VFQAoM-D0AjTKQXpwhx82DYiN5g,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,1vNFV-mDaLK92lJo0Mq6jvkCDzuwrSHT4lp1nB6KTzOCSO6comH5xNpEcvtZ_9pc0cQCD2kgDfB5xdxg0tAX2dF5oG5DHY6kxWXOBLnuAwiz1BkEFhw,&typo=1&ancr_add=1
> >
> > )
> >
> > at java.base/java.util.concurrent.ThreadPoolExecutor$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,SsQ3UwONvyPr9WW_l2zBPL-OwmRhL5K0tqgLY0-1J4ukt-wGoS0RPBCydXdNP8Y9HLJor2jcwQNr7eYoH6eyHr2nST0zn6kL1dCj0-XJ47ODQ1CK&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,aLydz0SGsclotqCro2upcTg44B_Ij7ckjApLu4B5JUwZKKdMo5z3RyUR4ikHFY8L3t_yB87Zp5atGRwoAPA1L9Mc9c681UjFrG16n-udVjeHtxQNxx8V-Nug&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,ceb67pKSJWHnVyXDhowqaqnw1emC2o1icbPvFlP1Ez2J9ErXw3aEcKlga83KIE47pZ6TOajqFHmkuHUTaNYJ98IMeYib2OlDC75rPTIM7IVtnExeYQb0CdEI9w,,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1mZX-u-XHEZ_e4Ut7YETRvWm7U_qkC-q9QaPeHah4K4yNl5foyM1VSExCtYiisknkelDj6_K2MUHGc65Kd-6-JG_TXLby0OPWodJVjX3BtKaDNBUzusgpzT6ajL7&typo=1&ancr_add=1
> >
> > )
> >
> > at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> >
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,DWbPBKB8qulo4SwULw5cFHdoemDsh1RZ6tdcphl6kJHY1lUU9R04kGYrhelrokAOssX6oqNOqq1H3Nx5WxGfsV89pNY82qtaQbNQC6lCQ2P9ZwHUzEA,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-nbEPDxI7J-DFvbJfVVt2Cs1Vi_hN5bw-8vQF3SlM1nthw3VbJLWmS1h73cupUxqBoHCImTzDqI1m5gutx3ZriLpLpU7IqutHCqZuT6CTVWACPtj&typo=1&ancr_add=1
> >
> > (
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,KyBPFkXAqcb0BvrWcVSxGnvSM_xbYzAcZQZEAaBC90eFhNGTrXQWNgyCh25vjEUGfJR5fpaEOBpqvH2Jryu_FDRi9SNh3gLyraCHFr8UH2b77vuH0_JCHliGFLU,&typo=1
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,-lUcdhDzaHWdAWLCc0IxWCFkKtKHxoEd64GWhzXAbeuG_Zm9mvn624BY6uS8MgQn-s2TMF8PbEPYVbOOy6BJuI8ByeiuAF9Y8f2I5wib&typo=1&ancr_add=1
> >
> > )
> >
> >
> >
> > [image: rg]
> > <
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,T2UV-1dji1RODRcU4Ff4j_IzfTH2n47BatzyLZcEOU8VROFr3GS8re7CjDcnnHCSaPEtR-35J3z30dnYcz5K66rVsMVxgDiyh0FvEDKnN67naHjYbn0NpzAc&typo=1
> >
> >
> > *Lino Pereira *
> >
> > *C++ Developer, Redpoint Global Inc. *
> >
> > 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
> >
> > lino.pereira@redpointglobal.com
> >
> >
> > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> > confidential and is intended solely for the use of the individual(s) to
> > whom it is addressed. If you believe you received this e-mail in error,
> > please notify the sender immediately, delete the e-mail from your
> computer
> > and do not copy, print or disclose it to anyone else. If you properly
> > received this e-mail as a customer, partner or vendor of Redpoint, you
> > should maintain its contents in confidence subject to the terms and
> > conditions of your agreement(s) with Redpoint.
> >
> >
> > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> > confidential and is intended solely for the use of the individual(s) to
> > whom it is addressed. If you believe you received this e-mail in error,
> > please notify the sender immediately, delete the e-mail from your
> computer
> > and do not copy, print or disclose it to anyone else. If you properly
> > received this e-mail as a customer, partner or vendor of Redpoint, you
> > should maintain its contents in confidence subject to the terms and
> > conditions of your agreement(s) with Redpoint.
> >
> >
> > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> > confidential and is intended solely for the use of the individual(s) to
> > whom it is addressed. If you believe you received this e-mail in error,
> > please notify the sender immediately, delete the e-mail from your
> computer
> > and do not copy, print or disclose it to anyone else. If you properly
> > received this e-mail as a customer, partner or vendor of Redpoint, you
> > should maintain its contents in confidence subject to the terms and
> > conditions of your agreement(s) with Redpoint.
> >
> >
> > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> > confidential and is intended solely for the use of the individual(s) to
> > whom it is addressed. If you believe you received this e-mail in error,
> > please notify the sender immediately, delete the e-mail from your
> computer
> > and do not copy, print or disclose it to anyone else. If you properly
> > received this e-mail as a customer, partner or vendor of Redpoint, you
> > should maintain its contents in confidence subject to the terms and
> > conditions of your agreement(s) with Redpoint.
> >
> >
> > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> > confidential and is intended solely for the use of the individual(s) to
> > whom it is addressed. If you believe you received this e-mail in error,
> > please notify the sender immediately, delete the e-mail from your
> computer
> > and do not copy, print or disclose it to anyone else. If you properly
> > received this e-mail as a customer, partner or vendor of Redpoint, you
> > should maintain its contents in confidence subject to the terms and
> > conditions of your agreement(s) with Redpoint.
> >
> >
> > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> > confidential and is intended solely for the use of the individual(s) to
> > whom it is addressed. If you believe you received this e-mail in error,
> > please notify the sender immediately, delete the e-mail from your
> computer
> > and do not copy, print or disclose it to anyone else. If you properly
> > received this e-mail as a customer, partner or vendor of Redpoint, you
> > should maintain its contents in confidence subject to the terms and
> > conditions of your agreement(s) with Redpoint.
> >
> John Lilley
> Data Management Chief Architect
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
RE: Recommendations for HA/shared-store using Azure FileShare for the PV or performant alternatives
Posted by John Lilley <jo...@redpointglobal.com.INVALID>.
HI Arthur,
Can you explain what you mean by the "inbox pattern" and how you would configure it? I've used what I call the "coatcheck pattern", in which a very large message body is written to a database or to cloud storage, and the AMQ message simply contains a reference to that blob. It works reasonably well to reduce I/O pressure in the broker, but only if message size is the limiting factor (as opposed to frequency).
Our experience with AMQ using Azure FileShare and shared-storage was a spectacular failure. Under our expected load, which AMQ has no trouble keeping up with normally, it would error and timeout when using shared-storage on Azure FileShare NFS. It could be that we needed to upgrade our FileShare, but it was already "premium" and we were so far off the mark it didn't seem worth pursuing.
john
-----Original Message-----
From: Arthur Naseef <ar...@amlinv.com>
Sent: Wednesday, January 24, 2024 11:22 AM
To: users@activemq.apache.org
Subject: Re: Recommendations for HA/shared-store using Azure FileShare for the PV or performant alternatives
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
I haven't personally worked with Azure's solutions, so just adding some
background here that may help.
NFS performance can vary significantly. For example, initial
implementations in another major cloud provider were very slow - but after
significant effort from the could-provider's NFS folks, it has improved
significantly.
There are many factors that come into play for AMQ + NFS. Typically the
biggest concern is the "synchronous write" latency - because of messaging
guarantees provided by the broker for persistent messaging, the broker has
to wait for persistence to "report back" that writes have completed before
message production can complete. In the broker, this is implemented via a
synchronous write to disk.
Ultimately, measuring the NFS timing and working with NFS experts is
valuable. As a baseline, testing with different types of disks can help to
narrow down that the performance problem is related to the type of disk.
Keep in mind, I can't tell you whether the Azure NFS solution is causing
the performance bottleneck, but I can say it would not be a surprise.
A simple "micro-benchmark" program that measures synchronous write
performance could be a quick way to narrow that down.
BTW, have you considered the "inbox pattern" which pairs a DB with
messaging? Any place where messaging reliability is a major concern, that
pattern works well as it uses each technology (DB, messaging) in the way it
is best suited.
Hope this helps.
Art
On Mon, Jan 15, 2024 at 3:27 PM Vilius Šumskas <vi...@rivile.lt>
wrote:
> Hi,
>
>
>
> just thought to give my point of view on this.
>
>
>
> Check if Azure NetApp NFS volume would provide better results for you.
> This is what we are using for our Artemis HA shared storage cluster on
> Google Cloud. It is more expensive but in our tests provided much better
> performance comparing to anything native Azure or GCP file stores give.
>
>
>
> Also, I don’t want to encourage to drop K8s here, but to tell you the
> truth we abolished running Artemis on K8s all together. For our use case it
> was just too much work for too little gain. We run native Linux VMs with
> NetApp NFS attached storage. For the clients cluster is accessible by DNS
> directly without any additional load balancers. This way we left failover
> flow purely for the clients and the internal mechanics of the broker
> without the need to tackle K8s health checks, timeouts, graceful shutdowns
> and similar issues.
>
>
>
> --
>
> Best Regards,
>
>
>
> Vilius Šumskas
>
> Rivile
>
> IT manager
>
>
>
> *From:* Lino Pereira <li...@redpointglobal.com.INVALID>
> *Sent:* Monday, January 15, 2024 10:10 PM
> *To:* users@activemq.apache.org
> *Subject:* Recommendations for HA/shared-store using Azure FileShare for
> the PV or performant alternatives
>
>
>
> Hi Justin,
>
>
>
> Upon testing our “shared-store” solution in our target Azure K8S cluster,
> we ran into performance issues, seemingly due to the use of an Azure
> FileShare for the shared-store PV. We are currently using "Azure
> FileShare NFS on Linux" and NIO for the journaling mode.
>
>
>
> Azure FileShare info:
> https://learn.microsoft.com/en-us/azure/storage/files/files-nfs-protocol
>
>
>
> What are your recommendations on choosing the best type of Azure FileShare
> to achieve the required performance?
>
>
>
> Should we look at other journaling modes as a solution here?
>
>
>
> Or, is going back to “replication”, with 2 backup instances to avoid
> split-brain, the best solution? Note that we do not need to persist
> messages for any longer than a couple of minutes.
>
>
>
> Thanks,
>
> Lino
>
>
>
>
>
> [image: rg] <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,5ajcMwinNa1VWTcDmA8Mj-nfQOh89Eiudhyxnn4whyIBqIOjlHqNbAhOJdyhfFV1IdcDZPCCvdhfcITATZ9P-w57BqMCkV6a4xW8PThHOaS4qFBBdk0a&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Lino Pereira <li...@redpointglobal.com.INVALID>
> *Sent:* Thursday, January 11, 2024 4:11 PM
> *To:* users@activemq.apache.org
> *Subject:* RE: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Hey Justin,
>
>
>
> So I am trying to verify our AMQ docker images in the local environment,
> basically our system runs as simple processes on my local Linux system, but
> the AMQ instances run in docker containers. Now, this works fine for
> standalone, and for the HA case, the “master” instance works fine with, or
> without, the “backup” instance running. However, the “backup” instance
> doesn’t seem to ever start up fully, that is, it is missing the AMQ221080,
> AMQ221003, AMQ221007 and AMQ221001 log lines, shown below from the
> “master” instance. As a result, it never announces itself in “Backup” mode
> as seen when run from k8s. I am using the same broker configs that run fine
> when AMQ instances are run natively (no docker).
>
>
>
> Also, when I stop the “master” instance, the AMQ-client reports these
> exceptions and never connects. Any ideas on why this may be happening for
> the backup, which uses the same exact docker image, just with a different
> broker.xml?
>
> Is there some fundamental thing that I am missing when running 2 AMQ
> instances from docker containers?
>
> My goal is to validate that our AMQ docker images are functional outside
> of k8s.
>
>
>
> With regards, to getting you the thread dumps, next on the chopping block,
> can I just attach them to the email, or is there an alternate path to get
> those to you? Apologize for the dumb questions, but I am new at this.
>
>
>
> Thanks.
>
> Lino
>
>
>
> AMQ-client exceptions after killing the “master” instance:
>
> 2024-01-11T15:47:15.322 [Thread-0 (ActiveMQ-client-global-threads)]
> client.errorCreatingNettyConnection:741 [] ERROR - AMQ214016: Failed to
> create netty connection java.lang.IllegalStateException: No
> ActiveMQChannelHandler has been found while connecting to localhost/
> 127.0.0.1:61617 from Channel with id = 0bec947c
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a980&c=E,1,--JOoCrTMX_jWzN7OBNpH51ACSzbmfM2BQMdiUEhw7AetHe2Swr4RH-2iRuMRh98ObfYPZkvyjJz8HJkHoLeGOihsqrgRmds9eKXguMxiiiaKmnkO_OSxj1z&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a980&c=E,1,9sVwvAhg-dCUoWlCTkknJyr35Kb1lORNDVxR6-uqw3mazBhZ5pG19VN92yx8WmUo86gmEfiM-fyes3vKIRRqmUr7xGxTepKfWp8f8YCxIKDeU3MjhJIbTwA,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a866&c=E,1,zt-CVldlmA5WggxivfZazCjgd3_Lj3OY6MGl5OVd2M7PDrI6oaAYX-EmNNsIpeiGNNIjrONSZlgN5G-KK63xVYQ-t6VC6qcluDUpfqgbGDrzAzwfykxIECMWMVc,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a866&c=E,1,PDO7x8ScbX2lkRczVyc7Qx5qFYh8jYI_QZSxIyD15vNJbt-HUSyFIAsL54Aas_i1kj247doxU1D-TF1f6UNtgifP4lOW0jVh-Ca8bmu8X2BRDUZyZYrlZd4,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a848&c=E,1,YgUinkp1CTnxOL4RXehYVUK6aUtheE1ND_UhWmJUJtyKxrilFc8vykkglXCRZUf77BZrtd0U_Ju0a4QgdspRI-oMNyzQh5hWQay4ckS0p4z7Ym0u-xvDhQ,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a848&c=E,1,M5S848Ei6ypG34y6nHCsxBxodJIiZs0WdncK5b_Paenx_yRNXYZToN-BG2GJ4M8V03LsOGQzIKSbFW7uKF8az4WwG4Xrzk0hIvgvf05YYBR-KraWq2Quc6cO&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1212&c=E,1,ZNIZLZvdwYwpL-xkz0pRXdKnZFOsDmCCXzrsdg-EpInUcQLSknJeZu2F_W9AhSILXnxoGmaxlYSp0jqu10gGnm2-Evg3rPp3uQJr8ruuNz4QPaTaKf7rqtOAJfU,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1212&c=E,1,Zu7gFKlm4HmSOURefrcnPu7_omfp3z6NkhUulfWOlvCON5qFl8o_m__R5jhaoNXCjQUWg3afJOkgibeky8zBPzqugdJrnx3mDcRvktMSa8mHPzzyhu9E&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1333&c=E,1,cUJprkrFcHupT11LWCziLhYS20HwCz5C-3yWeqHq3Kdl74IngbR8pUWLi9UlP6laatJYIbCKeGsIaYoYzV7Ii-u_aniwz4JNrK_eH-Bc5ZCx4w,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1333&c=E,1,QLuXIqCAvTMp9vzA3xlqCzu6uu_CAV6-6B3ZjST9QtAqqlHPPW031VktMnXrspcOD4IBb1JKvR2F8OzNZIi97M8NdBmUvk48uDHiFLvxVM4,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1253&c=E,1,tYT4QhUI236EaoHu-bd2NY-uVuJ3WIlZ_JS5Z3hpbZqfxwJ2A2_ki40UOEUt8rtEWNoPDn29f06FV-TbP20llGUE_Jz3rqMh1x5rbWHp99zmVjcgkhg9v9Yq&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1253&c=E,1,HGg4Q1mxe7PWvtXuKXcO1vTG9pGtPTivn_BSZ-cDFd_RB5T6KlxMhp2Z2iKrHRFFrcmAJyClu3xTgdygcYT1cRvfTIU0mnOfoNn73hij&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1496&c=E,1,Xi4IaboIvCsIgJLHmto5_91_7kVWXzu4pT6VOLUB_FckDqFe-MwAf3Tn8wFu6AkYEse2MgOz5PbXcpx1K9ka8jyEaNyac84vtqyeFji3RrKIkupwZsR9xTc,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1496&c=E,1,YDsqwgPbq8QPr_ODsKRUE5-FJdCxEgI2EdJdlE_ogGX_Dtkdqu0ZUW5FnNQHKMwQfJEjscTg0sXGDaY7D27_1M-BNdURAJybIJ0mM63IpSP_Kix0E7dJj0xAAA,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1074&c=E,1,GyMpBtXF0zJMlPd8R5QcrlIYrG1jYyvHdB-gSpPnT2nKuiKSy0ZA9cH3U9zea6PIldTV6cgJjPRBTTPW65zs8dVfN00ddgEgmMJAtuqEbo2eIxkpyo4_Thbo3f8,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1074&c=E,1,UaHGLwmlvYxEp5Zvya3OgHoGpFPLrtGO8817O18DQ_eCtU9hppVQz3ZSqMOZR4Ou0RyfDnJ25Dsho8qeOiDziqkIZ-8smasmw6AS-SC3C07T9A,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a959&c=E,1,yTfnN0IvPyn9DmMxZnOeU_hjdrjHupqdbiaqoD9MW1ugd7YWqfKUdlOO3cLcmKnLBPC17TzmKQSyrdmzpZFAvxZwKo8D_KTVwf_XD2SO&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a959&c=E,1,S2vAOnIQrU8itB1DHvtgyBp5ozbblk23_CGr3q-ohZ6Nt35WRISFB1PbqeDDdjRyaj8N9tzZSNUm_TpeEwNQg143NiFeTG18rXYlnq4jakQe2_H0HkGTrg5Qow,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a697&c=E,1,NTQDf0bLLSrVBiU5yFaEsSR3MzhjCm-v5XL-R8986Qx0bc_gFoW7JJSnOpHti-6JerM4AL-qUl_blORvOvDC6oMrroSKjN1fZh8ipqRK6g,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a697&c=E,1,HRBbQntC7regy9ruiCBdgTxHOjjHiqqEtdrKtyuFoRluNvypXTYG7MHAazXY4rr-7iE9ymvG-ccC_EuBMTBG5AjelMg7KfPvqiIaUkF1RfMIvoeiA8o,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,mQIMsoPaKSQkZ1VRg0ODHorYJuAYyVgvZHMAA239hlc6ujT3xHjKGoOJ9hde44gF5Qlvqo7XPtgCHbhf7Ne99NM2Wf9cnqg0j5ZWA-S4ZNL-FdDAFeDb7dvzkFs,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,WwOwlRDdAEBeA1Gg0ArmZFo1it0EZxIt6RsmaWdNGx6wo_Lz4qKdMxIHy7U_9S5bgl-33cfGzNtlYG9YM9FZXucb7-SjGMsXWWRiyGWMdHRJ&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a559&c=E,1,1IlJV1_pox6fWvlQ5kvv87nkhrf5pk7cRBcMWahs1ElrDscVZtWP-843S1Kjri-B8UPQ17T5oH7AGcB1erP06BjdKHfBhjobugFk8lZd5IcqQ7Q,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a559&c=E,1,Zc7KWsYoeLt5E_P6hgERE60yuujJHXiWpuT9jFjj4gC367dihQ6w7V22qGaDAek1JHPJHKEsYT6vqwQAPm9GSln6px2RaJkIttMBpwOF398rKvWxlzAOmI4,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionException(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a425&c=E,1,vib6Ga2NryGWCm421S_h4Yhos5O7OzRgO5N7LIGZ7BDQQCP5P905eptMajfgBcoR02KRoaVVKVb6a2Rohuqok-qqh66Ev6QY3YHRQBEGCCYXJeCMxA,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a425&c=E,1,BFL6YN5Kxy06vezMnsiL2nC8cdKvlemlxhvuHQ-FuWkoOzxoYzQuft1CeyeiQmEo-L40YbRsfc6dzmYWbWpQQb_xVyaLDbql7YwiFlA60a1xj1PSDmU,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f2.run&c=E,1,7rNv2g-TCyIHjGGhUjbsmsGERf2K_-zsztZJLbyf_SdTRk5n-UMNgBFbJrCJLBdDFKuxzsowV8MgOadI0DUa3rsPmPXvYpOOQF4sYuxTcv4DEE7XStsiH5U,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f2.run&c=E,1,sMyuWkYLp3LX6ibrSEXRcAezwbBM7qAOluPp_GC1jVesMs6jxDzxY4FFM3VjeigvvheAyp3Q7Oe5ACoqft2B-abCTUa6WF1bt7Uh4xjE6hHhWxo,&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a1252&c=E,1,fiJJEVQ9D4XMI5XRdW65UqXquUyRCjYN3ik0ZNbhPCXXCW9ZqpK7XrYloNZKK5EkwjhRbeLdUMF4JeAYSnD4ncjdjnXsIgTq0i-vboU2s_YK&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a1252&c=E,1,3JJSPZ_KDZLXcsGld2w64FmGt132PiKafjM_G1ZXb-ulOdS6Dwv6XUv8QHJqHN6CKkClQ7lUkNnakzibZdks8vAdD3gLbxCOyjzJGO6K745CLD2v2F7UK5zJSQGG&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,2gZklPlcwSiA_VxD57juT50iNxsdC2F7POm135-D7q0KCbqhG2EOqlNhfP57L7pOn4wJqx3QSIs7Ws5LySS4nMJM-6CMVsgE4GYir9A-tRdRUb3HC1bb&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,bpAKAylKI6Drp540buFl0pkNNpNnX4xnMjY3zWwpNyF_392duOqGQPjpdfCPGxrpBvc7_eIyfJpxl4WuufWnNYLFKdIg0Nu9WDi6STLxtiY0tUhT7eeO2iA,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,zz_PQsXSiLhs6hN6dCoMVr__-LNITICfu9whfe7T8sNYKafwJbPx0ydeJIpcOHQlV8KoVCsoJaHdJRvakGUgTlhxoo_r9nBSq-Q3GuOFKT1n&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,CqAtDgvRoDEnuFjXXv-RSPQ-NcQ5-m4IRpyQ9oiN0rhM40sqLFbhY9BMbc0HKKA9gNrRlaFd_DPaSB11LNOv4Rdahsd5fJLOqXHsZnvg3ShSJw,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,dOtVOpaebNETENjgoNxLOfnEDTl6JXfNUP8nbLUn_ODpPND1BYRdKY7SYb9wd6fnEHv9tKLhtIHJbbpFzgNPaexUYI--qVR5RFenA8fB1w79&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,C5wuD-LRtJ7605p6qERurgxxNLnHBKbC4B7VE9lfDl7f_JsOdDSkeiyMkYACWnhSckLFmjUW2eFKZawLMNYORn4gTtULbgzlBoD8LWUyNto,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,brUqu-rtXT9Jpi9_aPmOt43TyrXnEcWhxFmhTNjRzBGNXk0PNWWIb-ySKwHF38aP2sNKMK6GLK8wLBw8Z-3U2dxR-WQJEij0EknUY1qQKDzdh3Z3&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,rMwMgnKWBZ0Z_t5VCFvBOA7J5McS_4eMRlTXeB6c0yMri_CRxIy8PDWJRjrRtRMI1QpqaQtuBEcG4sCoeTP06BI4NmwaneyL2iZ_EcqSA0sDvg,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,0UKW5rRSlatQq443HXmOgPEfCWsEUwzEHwl5acWwEYeN53dHo_aELmkaqio3IDruNjKD-akvwKbWLieB-5SQVagmDJZVxIdMkF9yYvNZjmoTYiaKfQhkErM,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,AmU9mOKxf3Tw98cD6iOKGEUsELr-mALW0vP5N2eC_TxCQIG4d3rxa4xf_W2vXyThIPw3Zmp6MJGpD2c95V3s3JefMOVZJGioR7UrvoKjJHS1Qr22gQ,,&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,m6p9unJlLjm54pezttml4CXXKu86YpKPjtK1wRLPqDvGKX8lb1RF1m2TgYn-nv36VKGzC2Dzm0KhFF2KaR8C-Hw9OQkZslGaEKc2H7DFBDDYMlgRiA,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,uKLItaPZw48N90kGuOJ2giezmgZl49ttEaMZ2H4_Lr1mqtycK7JXSo7FWi90G25OhrIF9k-OxzV8LaDSYfMSQPd4uEksOAIsnz2-SSjTNFctPDccNQ,,&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,VSgBOyvoltdkUmo2D1AoPWHOPa-wHjTwCUcCjPpeQdcHexImM4wlFHVXaNKVusUmSbHnK71Af0QNmLJmHeiKW72by76jvIUj4gonFJYK6JyGag,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,gIn498Y42-KQL16Kw7akU8ywkXBbJPMfxg7-Ivrxphmo6NJdJqsbcSdPEAdkwVYS3FgjewkEVfwxLXUtcBsU4QHUbsEaVo5060hl3qEBmZeRmUBspRNTPB0,&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,cSIXdjAwlzhUxm9mRFMDwkvvM6XLO7p6KhGq4WtJI7hrW4tImZ_1CrDPG-ANNyvBMJAlavuv8nLzVesnqH4efC_PzkQF-ceGB8ZGKm9A-BV-&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,KRj2MYUCdG64pKABEvud93N21Qgh8ET58T7SDN4ZzK9ErKl92F_UyzJtq4y9eDIqY04euQg_cRGOTddiPcrePTpRRyy9U7AEk_WKzWUcU_VDwdQacLRCEHF4j6ag&typo=1&ancr_add=1>
> )
>
>
>
> -------------
>
> Log excerpt from AMQ docker “master” instance, showing lines missing from
> backup:
>
> 2024-01-11 22:38:54,844 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address DLQ supporting [ANYCAST]
>
> 2024-01-11 22:38:54,853 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
>
> 2024-01-11 22:38:54,905 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
>
> 2024-01-11 22:38:54,906 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
>
> 2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols
> [CORE,MQTT,AMQP,HORNETQ,STOMP,OPENWIRE]
>
> 2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server]
> AMQ221007: Server is now live
>
> 2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server]
> AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.31.2 [live,
> nodeID=320ff011-b0d2-11ee-a54c-0242ac110004]
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,WIQnILFOGT12V7T2V5JgdPTF8Y-yZZKUo4gSIgdk9Xb0RKhobbQh-PjrSOCrn9J79hRQkhxWG4u7HGnntjzO4n-TgjsgxAzGPARQB9OM7Zk,&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Wednesday, January 10, 2024 6:57 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > ...I’ll have to instrument the code for that, right? Or am I missing an
> easier way to generate those on-demand?
>
>
>
> I'm not clear on why you'd need to instrument the code to get thread
> dumps. You can generate thread dumps through a number of different
> commands, e.g.:
>
>
>
> - jstack
>
> - jcmd
>
> - kill -3
>
>
>
> You can execute these administratively or just set up a script that
> generates them every so often.
>
>
>
>
>
> Justin
>
>
>
> On Wed, Jan 10, 2024 at 7:43 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hey Justin,
>
> I did finally get a chance to run everything locally, and it all works
> smoothly with failovers in both directions and the AMQ-client recovers
> within 2-3 seconds and with a lot less log noise.
>
>
>
> I did not get the thread dumps yet because I’ll have to instrument the
> code for that, right? Or am I missing an easier way to generate those
> on-demand? This is happening in a k8s pod, so not easy to connect with
> jconsole.
>
>
>
> Thanks,
>
> Lino
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,P1-JTENl1pkpwfayJeVxTcbmyxoXqv5aWc8dgoVoQa2P9YJ6ITYanSwPPWI3OOvd-1ZuyvCfDdUgZSHNKSzXPNMT51ZJGff0MJI1JNYM4KnicxOXm0JmVYcKXz0s&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Wednesday, January 10, 2024 3:02 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > So our issue seems to be only for failover from live-to-backup,
> consistently.
>
>
>
> As far as I know the client doesn't make any distinction between these two
> cases so I'm guessing there's some kind of environmental difference that's
> making an impact.
>
>
>
> > Is there something that we are perhaps not doing on the client?
>
>
>
> I don't think so. The connection URL you provided previously looks good.
>
>
>
> > I will extend one of the example codes to use threads and sone of the
> AMQ-client wrapper code in an effort to isolate this behavior on a simpler
> code base, as you recommented.
>
>
>
> Before going down that road it might be better to inspect the ostensibly
> low-hanging fruit first. Did you happen to test everything locally? If so,
> what happened? Did you acquire any thread dumps during the delay as
> previously requested?
>
>
>
>
>
> Justin
>
>
>
> On Wed, Jan 10, 2024 at 3:04 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hey Justin,
>
>
>
> After starting fresh with version 2.31.2, the examples work out of the box
> for me as well.
>
> And, after ensuring that version 2.31.2 is running everywhere, client was
> still using 2.30.0, and refactoring to using the recommended shared-store
> HA, our AMQ-client is still behaving as before. That is, failover from
> live-to-backup, client recovery takes over 1 minute, and logs report what I
> posted earlier. However, failover from backup-to-live, client recovers
> within 5 seconds, which is the expected behavior, and our system is happy.
> Please see new log excerpts with more logging included for the 2 relevant
> threads.
>
>
>
> So our issue seems to be only for failover from live-to-backup,
> consistently. Is there something that we are perhaps not doing on the
> client?
>
>
>
> I will extend one of the example codes to use threads and sone of the
> AMQ-client wrapper code in an effort to isolate this behavior on a simpler
> code base, as you recommented.
>
>
>
> In the meantime, I thought I’d ping you in case you can think of something
> obvious that we might have missed, especially since we just moved from
> 2.30.0.
>
>
>
> Thanks again,
>
> Lino
>
>
>
> Our versions:
>
> ARTEMIS_VERSION=2.31.2
>
> JMX_VERSION=0.18.0
>
>
>
> Logs from failover from backup-to-live:
>
> 2024-01-10T19:52:29.588 [Thread-24 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-backup-svc/10.43.229.161:61617 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d7ff7885, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,Mr3pbdt4mcmFaD997_pq47Opt2EcMvLLrsHvGYc39vqgE1v31obNH7GgtocQkwJ8I7NdstAMkov-HWYt3HH43x2PfQNR-mmb8DU84BffgA,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,O0LJ4VfRCemWEaMLULajHsgUo6YseAVDyBlrTm2G50vM20WhlTUV0ueY1DxMRAMStXAoc6wy5Xx3x1vi2XsnE5F7_iNbL-MZ03M4IS9G&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,5uxOr5VVLMHc0sdf8gtHE4s7BtRHg-Mv9J8MsQZwLB9fZHgQIL9-gCiBOnBZ4AFw5etlj6zKObFUYD4UDlTyxJ_Nnsh0-ZD_zKuNa9TYGWXoDBciCXBTXfw,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,VdcejKqF9B67eWPDv-PP_0gR6PEmUxFYHHtW8kyn3Gzg3q-HHzL-Lr7oztjj2LQumPJWO-1KF4OPavU8c3DW4zPgrUsidhnnCpoMUUn_&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,esKwX1OYT2CmcBEJf13EtgEtM5bjfJWvy3_kNngn3FevdhJTot82KGD27QyADUHosT5nnuoY68T0KvZtSMshB8CtXbEo4iuodyuAe6run-Yf_vtpdfZb6iY_jw,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,xKCDuAN84vJBjsCp75epCz7V1lR05TFhETdvpPcKntBTPnQk0zuuQSei7-ly1RPUUsHDP8ZAe-IevqwGtTGfEB_yNzw90QcOpEx5BaXOW6E,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,Em77Llz_l0_fpFOmT3j_K8dNPtK-V-hg3vcuhVhKANwZIdL4YZBxXt_ZqdNLQ9Wjl3LjXF4dODyvIovi7TT1_YjfHVyf0-7hNALJSTea4D01F2WNgy2JQURVGzkZ&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,pq7BenEb7b9i-FYz1LnTy-gHYDdV-28FiSv3pkV_i-1TQZtRxakbT-rOKkgLgz2IDx875Dk5EE_OrwsUcew9TIsvrcZZZ9ZIlwboxMEq9xnR5odpQL8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,mtE6QufgedGCjjeJ52aBLxixFnx_MdeBZW_SEsMGLo69xglp7tW1gUctbMB7puEEJQQ343WtJ5VXehpalJB8CbzpSMkL8xBDtis2B56Vr3hkehfBrl5Y&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,8mmrZ5alRiVlyEshqjlGpedpt7TuUVBJOy8DDm2qHLGoY6v3pR76llV59sMsYwy2v4rwLqiD9Rcf3Hg4QSCZo7rdO1rQgMNapCKji2jg7T4IJOk3OAeLnIs,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,ASjiJy0t1SbKglekpcLqYuAzRZ7xPKk6a04qfGkUnOjLxrlSWELC0E08z70MOAVeMF06IjjCFZSpnFMLTpZ-14OMPdIRtZymzcHO9TnxVWm3fyrpiQ,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,RLTYzCoyknTJ1GB9szMq3Jg02QAnt7juedv1HCTYmmA4JU1DO6MlJhkvzOTBqHbeeOdKSvwkualBrhGt0fUk7Nl5YLzlgfepqTWcDiBmXQ-_xSMcAHPw&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,iy0sncbqed4Eoa37FJyaBMZ1dBdf4bdmG0oRJM4-hoomUQiyzi49VCEaJhIjYL72cmZ4An1QpH5PvNS-fTen0fShUt4Ft9eGa7K36Axs9d4,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,iYtEGEWoGf58Vq3CGAxBrH3iWZ1jWm1N1b3AOFPNlx0kCahcxHEwXT7ZAQ_h3ynXTeKJSkvmBiGUkbqE3bH7tGw-Gn8VV7R7LaCkxl9Dch5I4kQDfJDKSg,,&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,FfU9MXaLianm9R_iutvVZro3GOENZby33tX7xvKXuWHPcjQQFRE3AH6Jn59gGGRXgpSqQZj44nrnkXXiZuJF8dyFHp2yO0m_E67lXB2wCQ6NFKE,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,kH54RADAl8RcHDeCxJW9b-P4YJJFElj_Nbh6mHtmwlFdP2tKzzbK3wkAXg9tVqhIlsbd4gr2LJkn7DWZ4I-J8WVE-7UovVbVakuQTTxc2pbR9vuIDiEFztc,&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,20ZrvQagCRF4QTYRrJ6tsdMTwZ5hQ2XtPVPpdtR3XEK9tni5a5bBz72Os4-2LPRdgsQssQTFjXaSvbIyReL98hIBq0LJ6Xg6u6NQO-VqpG0S&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,V9-XUYgB7GoqPyGhEdE_CmtNood-yvXw6P9wRIqQOSYSWuTfiF1bhonQejBQfgJdHq8k2TsE1RAvcSOp_WzBE-ybl0l6rc5pIuMGdC1LUITpdk_Iuha7urSje4Y,&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,86MdH7Ho2updjk9VNoTbiFAZDkPZva_rcFIo_FQMkfNJuukTUa1ho-6FJSKBkrSDldwQIHL-7NAcbMjdrMtoIyL3JwtZRZLRtf-t1Sw7&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,RNgnFH7dBkD27TX353Mc-kqneOLcqFXEgpGywxg8VduTTYNyYFt5VqkKfirozQw9V88WOI8XK4v9dvrtPIl-3u7cp673GvKxq3L-XgqypuDtSnkRIYnh813U&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-backup-svc/10.43.229.161:61617 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d43c72b6, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,Pb4Gjw8cAEK4mRPSKvySF-TZaU1MIQLpfPuBIgcm6dUgwnlv7IrUo7JZPV5HWrp50EnTgrQLowfSia4dVSwM1yYk6SWvd6O-FGU-JTKn3G-XpQlw&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,L97BTn6AFLbUpYgHzAlQm9pDJ4Sn--TlGDpz05xOE3xd-qn9krfLJ6JzGTaiSpQwuVjuQ8KUPfUSW6N18pgw44-ay7eqYb2h0p3sxehZeXAC&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,SFBqyVG7Ebud5QIGYSfD2X8qDRvMZ2KUna2iM2x5eti-y7lgMHVrQZBEQFwuW5LKaFuAc9Qy3LSF2WfNwCoavbGXEnp-fsqZmlWmk_bojWc,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,PiedcDlDv-nxD-EF13k9j-sagD4clTOVvCs6xyDQvnrPzDUxc4xAbnHg0d7hnqeG4cVJZaO42LJvoNm59E3SmClgLAft-3N7F898HO6Obbkm&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,iAzsBKgfMT7O0K2WksIFtRro8pwHur_nVf2Mxxf7qb9X2As35BL5LNHfI7mq4bj-hNtMbAUEmNWso5KnY69nZhYm7w1g9cUtuHY7o6wNJIRaF3Y,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,G5SZ7tGYpGBQzNRxdmIzsp4hWlUsmbY3_881k0N5lFoFBwuHt1dMCkKHI8wiZRGpS6qDY_UDBvDmKVQNRIHXIIyg5Df2nnWDCTLnbvWCO20fW0PC&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,9IAw9lo-op4sAvrd7fWnIK52Kj6kbL1vJXQDSwfvAFWwNnWVzRXRVamDdm8qtwAyEqxehrhyhsfWcx_l8vRA3RZ2sdPZlX4FYOGgLd73B6GWq71t1BKwm1c,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,MsagygznR7ie9qgJLlMogXq8ex-X4ST-fg_vbVbgcBanXur80DVykS1ftnzq0e4bWLE0k8EiNmlhlezemqTlBEncWMJz-AQAKdTxEk4WyVY,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,uWuvrZ5S1eIewMe0I_OPx1KRnyK_EFNo7e-8kD4KbrQEveKYerYlrcyTZRsU8tiZvDCsSC6cs02SBC7v-barc3wMFibDjfG1DSPeiot8HHozgiQ,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,n7l9CUpZZbFH3ca7Iu2wxM3kNfged80mMRcu0Vn9wIoZz7aKzCP3c1OVR73iqF0oLbir7CjMHKAu3sfrR2cLs-EP7orqDWQKQApkzTZra9CJag,,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,WquhQ2ZrbyUtmzRk9zvDtleE6eDakAawIG_S7GMWeknBD-w9ubvsR4JenMxQRC_LRIHmwoDFk69sYRfGdWud872mwzcasuNvSRpohGjF&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,PePvk640nUsvwJWNJY39J2R954rSgiFVPPSQLOsjYskMxtDnDVBuzQiZ8ut5SAPoqzV9uYW6GX3ekOAAoou9eu6VpHa_oFc2-7d27W00VBLU_o_Zv4wEUw,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,WKXDaNmnUkiw6yQLU8DeLmCsUKQ02DYgVbiKdQlUJVDtysFCse7EIVda1biRsHIMR7fSPKb425g5EMmgZg_WqixOfOrCF4AXk3iBNo2kL0k4pTOcB4phWg,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,REmfZe55ytZRsqZvFkCNsYUcRO5Yd8PZwIlQfGystV0qUeF2mXSYaWKgleI098Ac40PND4ilF-ukqEWDY8N04AIRQgGgKGpG036xx7jNSc78YS6UbKWiMNUHfNc,&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1_QKSgioBO_0oJhMZNHydhglIxieW-0-3PKhoKaGwfuEJEpPYoUGcTom7KYsAwd9SKfzJvpaHOV7mLeBqqEIz2jc5rhSvLMO_K1wHuRr&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,nnqfN76WKNs0ZYhVPzea17QO5aWkwmLqbfYrmQJ_qGQIgTzCL4PSXhjU9jy4mhbCUfl53cwF4ISYabr_zD1GUvCgv5m4dYW1_OROCBpe&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-elXqIfEcoGJrq2i3bA_kciC1k1tdjh_d4QwvDsIGDqWWE90FQXTc6Auo7KAvxz35dJB0xa76aEDYpl2pOm3UjommGmjsTEoJaLHASKfMw,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,vlJ5vYZAfrlVp9kL50a_jwWhqVW8-LMUgzeBb6mn-udcS_fMR1yld889bXgOXTMDiutFqKYX9XPh0RtxeXRLtnN31WqeOlH_CZv4_ZyhnOwXvm4J&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,6vJ101x4RKUCOktPDq3GW-EUpGhMX8p7F8zG5XtKtQWXbkL0x3IcEVLTzeALD7hlP83H_AoEGKTHtcsMiJ7PJh-GGfjxoXx5-EdWXbVHBLOE4w,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7Ba-VAkMm3x0E0f6F1f73TH_y9Ylo03hx9DEz4kqWzvrXy0gAv3D9zIM35eXIaB95UFxo72DkMpb0nKW_a-BIF89JobbRqjyVNLdC5K3L1EM520B&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:29.592 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:29.594 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:29.593 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:29.597 [Thread-6 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d4cf19d5
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.600 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 7b2392ac
>
> 2024-01-10T19:52:29.602 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.602 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.603 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = c1fb4c82
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.610 [Thread-8 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 06b7b3a3
>
> 2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:29.611 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 65680be4
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.624 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 97a109b7
>
> 2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.625 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.629 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 77b2de2f
>
> 2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
>
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:30.612 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = f92d7cd6
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.613 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 1d44895d
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.614 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = e8602acd
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:30.631 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 0a217bfd
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.632 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d2aa00d7
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.633 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 9a4f639e
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.634 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = ecdb1cbf
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:31.000 [Schedule-Runner]
> SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service
> as system
>
> 2024-01-10T19:52:32.001 [Schedule-Runner]
> SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service
> as system
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 2/200
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:32.116 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = a6b3d488
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.117 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = c0627cd9
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.118 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 71523eb3
>
> 2024-01-10T19:52:32.119 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.120 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:32.134 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 2/200
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:32.135 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 68efd0dc
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.137 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = daddef73
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.137 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 84813ce3
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.138 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 533990e9
>
> 2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
>
>
> 2024-01-10T19:52:34.370 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 3/200
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:34.371 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 1d8df71b
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:34.373 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = de20e193
>
> 2024-01-10T19:52:34.373 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 3/200
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:34.390 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:34.390 [Thread-10 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 4661fd5d
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:34.392 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d1c14a74
>
> 2024-01-10T19:52:34.392 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.404 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-10T19:52:34.405 [Thread-9 (ActiveMQ-client-netty-threads)]
> ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up
> backup config = TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc for live =
> TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.405 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 38, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.409 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-10T19:52:34.410 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 37, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.411 [Thread-24 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 10, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.411 [Thread-11 (ActiveMQ-client-netty-threads)]
> ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up
> backup config = TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc for live =
> TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.413 [Thread-24 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,PTh4iDktbimD61S_2u93Qof0SOatHLPxYuhTwx4RmztxIbGXEw4EVaQT1sQ5UdxKH-SEHGoaBTBlboJQju_O44SFZvaaWLO4-MmXleNiYsXze4oL&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,S5999ok8Om6JjNfBLhC0R2hYdl3dGqPWsLpis0MTx38AhKXPNqMMlH41RrDz0UZsxZzSZbLvUEOYFEa7hwM4N7gAZfCS9udcqSmOwhLg&typo=1&ancr_add=1>
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,GIwgBhyL-Wik9l2AqNI4MjES1gdmJUfEqVrbtU3wLgBtGl0NlhyMc7o4-4Kl09iahBVAsbPUdhwYPs-RPeg5-ZzPPYTCAeU3rPM9jvmXg_xqUP9AeLsu8AXKnw,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,CF5ZPcR0aBH6uaRjLjSeWoBKLnRwUOOgvrD5MzWuerKGIs1FEa5vZF8rdF1kWSKbLwGQlm1IsOeyey0f1t7eiA1ApUgs1EDhL7ilSV0c6rFxEanucXI,&typo=1&ancr_add=1>,
> needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,h6ooxFeoDRKOe6DMrvyxN9DvLJ4nTkX8-TCIoHrc-Xf9WcSsal_9ZcG5nby3OeHUxsCLiFlHvYoiGBCvFWFPVCT9MCY89GTMxBPVt6ikH_tq5aCUbw-Af3P5Eof7&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,nl9VlpvL5Lx7cxWUON3eMxxXV1my7fPq5WQLaWD1ozszhEPr1wVfqCwFilScj02MzRACloIhLZ8CgyMWoa_OqCu-Tf4folOdvKvtooRq&typo=1&ancr_add=1>
>
> 2024-01-10T19:52:34.415 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d7ff7885, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,heQgfr1DTWeXz6rPEBmlpzRARyTcEfhTUHsO4tP_oqHC0157GZ10xCh6kckF7RiFAWYwhntDKaSjR6taVznn4QO1BxPmGpIzx9XNW_Y5CrSGq3kF2Spq0KadplA,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,awSP7U_0_MIm7Km9lwe59SrlpKsd9iUMA2y266CCeNZS6GJuEExdDYVS6EVMnlmPlVN0W1aqSx2Rss2Dm_YIM71s5j7OAUz6pAdGaTqnU_h2RyaoD6KyuoU,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,sNDACy5rBtvF3taiFbSF6Lwj-3S9xNTRGgz8biYJnUVv9tKyCw-XOFDaeS3QCKCwEskbKsJK3TsvKwTjqKClYruLL3tdaQ7Q95FhBtt9cbjs8gI,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,8gAKfG0MZn0OMdhCHJ98evOezuGoCS0mNbGWplSwawxhF0ed3tPq9vKWoWUbn2xmoLJ34skkCTpI2tUuevSCeVNT00tNpFlOJlyAjGWu&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,OmCd-aJ9EwCguVqoTqWVTAotLX0mbF8oT3l8FWGQ5PdepUgkFuHDli-CBGY_bflWpOE4vCbvOC4wVmR6NSSH-SW7eDc29yrDo8pqWHwqE5AByEQ,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,tXm2hjddgUkJac0QpMg4nVt2gwg5e3ORdql8K6xsJlBZIxph-NuPt1Ks2iB0ZarzoFELPAuAerVMiQpEU3nkq1JiqLAju26hE6Y2c3MwmKRTj66O8tsfQ-WFSh4d&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,T33i42EbsHkh2YDdE_TUPZeZVywUqlNZnmvDGLR9afQz06e5CEMug1KrQHqXcgwdVbJwb3N1U6pP9VW_c72K7vptWyoU95F4RuMxp8CI-ypOIw,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,P0xsXGcezpL-KzlLSOt_rfr6b6RF51B2hthJzXMGD04QeyLyg6AnMvF4M8ZOEwF90ojRdT3CtOy0x9HBSpGK_mmF_6Zl7Muz2TKrZRLAxQRxhV5XnvYlwQ,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,EkDWiSpN8nJrODSKBaJDEOcA-o_s5mt4eZAW-Yye2VQRh8fil0VD85DfVJKvjfhNZ8mbuniJBnUkN5CyozCceb4vFqbrM3KQ6brg_Kg64w,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,qctWvKkM1LAyegiROEl3hsiEN3dPcBcH4jlkBConOP-G2cgGl6X6XG-kdeQuBBOC8BmtTf-VmlbpJTjC_wznn8I6g7pLOg1uof2v0j_r3yyK7U8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,Ls7Rscoa5F2LNCLQqRNaI79Grf_Nj8W0C5zHYWcLXmlhwHE7D0qaQpynpuVb41nbC88w9A-BIDeTNWxqKUjhHXAIyykw4kNA1OjyFnMGiAXv&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,KRtNFryEe6jhqdT3of2FZC9PF7zGlYTxJ3KPFwO6TViZkyaHiybwHP3JmOhQ4F7yZtorUVRe5phyCswDWLc8lIFBVowIJygjwxLmO-F3xJSonSQ,&typo=1&ancr_add=1>
> )
>
> at
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,wTUvUO1DE5J8AJnZBpatxMFX27qqm-i69bNBA8umuqKFLjBUsYeoJRa5hda8r_tsIBFQIJVLhus2JZPlBoDWBiq9yEqsK9P0z8B-AN1Bw8HMFeZy&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,ltH-LxKKSiKev4cs4CgS53B1Sn1z2t9IkxY_vjCKeoXMalJfF0yTujWPogG1a9qkWV5DrAT_qCTk7MdQa3LpkAae5yHTe_19fREF16p6vMMJqGXqLjw,&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,U2u9N3XZe3KrRAFV8FC6A0jb-i4WBpO8GabyKN3J_ODIs03SMJ7qyEFVS7_JKTUE3DJzdoAE8Xe54QzWW_zirqAVS2X1Bdg-Jgb0WJY72DgHw7U6zuGETNTb-oo,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,66enqLb5-TVz_cB0ZCkDW4SpX4c3v0jslINMsxhHSjXfBWrIFoM8PxfSS7_VWmc_MdfmjPHIKcLm4zfcCe4MggwEWLxB-RjiUMqnmc8ShDJLoit5&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,8HE4vq7eDvOR4Akgo1D2PL10SpipE8h_Y8GjSHFCP5Qde-w7osTJWj6gmORajGL_vaB_k321Nub6ghLN08IpaSQY3kUUISFTpTQtIL_sWVfg1z2nup521pY,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,xM_aKOMjg39ORE7gTvaGKP72ytDNvt6tjanfNTzekduYVGBvn963oijBiSY1JOQeorfgcCf57lVZSYIQ20x8sOFAH5zNexuTI0rBtrkDrtcJpq-S&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,8KL2V5Hl_UJmFqzePqxt1m3TWIAHVbzvkIeRkpBBo4pbg8otQlW1R42WImcT3U7DhvBAif2arGKYqt2zg8k_ife9D8XXoyhw1HWfRlp8IoKyDxSJwKzrLQ,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,Jr0OyBaxbJX-r7nzGvHtUkNxDkYucNHeuCPW_Mqw5I4IS0dve_XCjayhv7uvFQTdinHTsHqf2ZtZAn_qz1ZnE04PEsE0KWGfHQUj6N5WIKcNhyQ,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,ZPAthbXOO1oRXumH4hVlQm7THB9GXz7u3roG6FE_qLs1OzYIC3kGaWF0EqXlBbqhaVgSpZtFqo4zgsGHTD5ZMzLA4uhu5UijndwmxkwE4P1qqBX0A1rpTm1hAw,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,fTod1kMxyNrLe6wmpPVQB1vco-7IrNIs9Dx8h9kWYDCH7xYH8V5kj2iao_X4D62DX5S24Xi08LA-xy2zHwj-LIzSKOA38iBBAlHdMD7qu3lK&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,iYpsZX-jjnbMrsS1VvCBeXnN4ZoC31VLIPM4c1i_l5RmG_0_FTxKLBOSY36ZxeAHJYyQ2hjrtS6gPKfbYmS3x0upQbuFRCpQQ54oW5CVMcUeaDZKblgP&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,37VGZcLM1KBT3tttPbLrhK822id0s0HCTimMYb18Jk-8bCbWxvPl2N9_yFFLgXa5pC9DXbKX6c8DMHolUFme8KRWVQdBx_lbRwtpLIOp6O7uiuV0fttkzvI3jxdO&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,5BW-BLxrERkrSISvOqkdNRbxf6b8De8KYd0krQIONwx_IQ5j0CQKg-8wJJAEVvmiADESxORcbca8kwQbvkQ9gCUsFpaT5SXFs67USI9Bj9g2BWJJ&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,zIgfFXrrGxOB_lKTtscVc_jXQlq_pH7TMnaK0IkixM4MVqoSgc-Wp5VV-QGQWmUtW5dggcgtYpRa7OoRHCBvDew7fxYJq28eSZVguCiuBiH9DzdKTHAXC5o,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,RBf0UlvPRwHDakldKITeDilMpRb7wrv829yGFB0b1HT4-n1J88dG7WMGe3t1RooyQQ3kGGqr7TNJmm9Of-ySWBfkV33mxvs-ZgzZB_eIScsIEQv6eO07&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,b6YutPqRZaL3jtcBxfqj_SiU4M_x_Qx-GeSmBfL7c6CG24IUL7AdjJxpyosXZtCtKco4bSDRxOj7HJn5CWgbgbPzJ0TFuXzJjs7VRUKVhg,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,3L8S6-PUyDPeFVGKFKyVjd26lXnTihav9ECN0TZYTsG8KgSV9bRAVwWAJZMpzImx9pHWaIYywf6dJN7oqN0-Dr2iLzUutZeHeC4dyhh0StMIshBrIU8,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,xdw7r1BqGJuJPUUyo2a9wj_y2KKTC-j8c0DAocpHKzpZZaVOgPcxdng0Xi_JMKvlGYmcUvlTfZMm2FYQ7pTrF20xWUreNaRQqtU6rlWB3kkuPmMNBALyTBs,&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,DO_wUMKwP6osP-Lcj9Tsuc0xOfgXkt-AeVu4ULY02dHEgw5EY3nKNphpGPqFO3YW8Q8v2ha10FYjj1eZSEk8DnSnWAvwZqqe5UTDD-hIo-a3cjd2xwk,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,4iCAaiDF74c7cCKTdUIIlT6Qu6BmbQItK-gaAIrQtpNdie21ccXNS_dCydALepUtQNTgsVsVUs1NavHOugg0IlFK0CjTj8TKnThm55ffUeOx8zHc4oZUYpE_LRsz&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,SSiNb_ntMVKcpbykl8o6Zy5_dJ0Kox8JfbhtZ-GWRQBXEq1byRJrmjqbE-m8XsYk2KCZjykhzfVKFgcnYdQwoH4uNy9RUFLMCgnu9prRsocj2HRywrm-law,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,qBGyQwtcrkAjVKJZ9cZzaW1ivhGQgGfhpcJUS_ww-vfcQVNfD9OjDwDX1XjpzJWYpo08MqSOrKLpgO8d72TBVtSGg2OQ4AmggCv3IUgKeA,,&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,AA_FyBPbdHM_iFmpEg3W2LJcKL5jBsndJh7xPd7S51ckhU2gWbFmFseljX5EVAmxJQFSXrnashRfZnghyhaA3dL-AiiLHsxSKV8q_xsSJ18,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,Mm6fpsoFsFLUtanrkJ6cqlk2y1zCJbPDIfqxsd0pS_sfS05oTAHBNIXIUUd4iA8HJY0guikOejZmykugHqKeopxLvJE3UmpX9YX1Y3Gu2ToK-B8aRw,,&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:34.435 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 54, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.465 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 98, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.474 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 20, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.483 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 128, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.495 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 158, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.499 [Thread-4]
> AbstractProducerCreditsImpl.checkCredits:184 [] DEBUG - not asking for -1
> credits on https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,PRaZ89srZpbRetuvy2A6q_H51waJMHKZwMIm19nHjdCbZxEVzh0MSm3wENcuBR3DqyjtAzHP_fuBYrRsG5jZl1sOJ6prXBGy7eOmaOc0pX9YvPQ8bDEN&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,BYjREAStKM-0e-e5OOY86CxL1g9WrAP01m5ROYtsGCMHTh3wox28vU8_r7Y9e7QwRiDtJ065Xst_zM3u0oFx_K7Fr-ZkN1GBxPR6aWEKrm4nQA,,&typo=1&ancr_add=1>
>
> 2024-01-10T19:52:34.499 [Thread-4]
> AbstractProducerCreditsImpl.afterAcquired:98 [] DEBUG - AfterAcquire 333
> credits on address https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,AVjYtSKN8IYY0tqPPLmNCShOm_eJRRCIMxLfSPfxXdm2DLLqF8eO5Kj5vXsOo8-8JnHebRNxN1JVuUNcmdVZr2uTToWlkSsgHzOe_A7Em9pERfdULzX2&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,r4o5lwz6oA1VbUpicHhTmEpC9U7BwWBMw8ZYMcrl4g-jVFykL22Cb2WJIZLOqv-hLbxRqgThYlmO40Jo84JsPHe-IYBjUU8WMQ-jy1lZqxq3YQk,&typo=1&ancr_add=1>,
> pendingCredits=65203
>
> 2024-01-10T19:52:34.506 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 106, performing as a failover operation now and recreating
> objects
>
>
>
> 2024-01-10T19:52:34.680 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 155, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.683 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 21, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.684 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 129, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.686 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 57, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.687 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 174, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.688 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 176, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.689 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 56, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.690 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 108, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.691 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 156, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 233, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 112, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.693 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 187, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.694 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 110, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.695 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 144, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.696 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 235, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.697 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 28, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 47, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 30, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.699 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 77, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.701 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 65, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.710 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 70, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 34, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 159, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.713 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 76, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.714 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 119, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.716 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 59, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.719 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 87, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.720 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 80, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 166, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 149, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.722 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 137, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.723 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 163, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.724 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 179, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.726 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 181, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 17, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 58, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.728 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 32, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 49, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 234, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.730 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 143, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.731 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 42, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 136, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 178, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.733 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 55, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.734 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 95, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.737 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 48, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 16, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 146, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.739 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 22, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.742 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 43, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.743 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 130, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.744 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 36, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.745 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 60, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.746 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 117, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.751 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 14, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.752 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 81, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.754 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 93, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 138, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 100, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.756 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 62, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.757 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 125, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.758 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 15, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 99, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 44, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.760 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 172, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.761 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 84, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 79, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 122, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.764 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 64, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.765 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 88, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.767 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 151, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.772 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.773 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 96, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.774 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 27, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.775 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 73, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.776 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 90, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.777 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 134, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.778 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 35, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.780 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 118, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 186, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 41, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.782 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 31, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 150, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 145, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.787 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 127, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.789 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 132, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 170, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 142, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.791 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 53, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.792 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 86, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.793 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 147, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 115, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 12, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.795 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 135, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.796 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 68, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.797 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 103, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.798 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 141, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 165, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 18, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.800 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 33, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.801 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 116, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.802 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 124, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 26, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 50, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.804 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 75, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 85, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 69, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 97, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 101, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.808 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 153, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.809 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 107, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.810 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 175, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.811 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 66, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.812 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 111, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 152, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 45, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.814 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 61, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.815 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 113, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.816 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 105, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 162, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 171, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.818 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 173, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 184, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.820 [Thread-15 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.afterAcquired:98 [aoy5vnfnjetz,6ek4wlg8c0ad]
> DEBUG - AfterAcquire 781 credits on address rpdm.v1.authentication,
> pendingCredits=63974
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.user.group.query
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.user.group.query, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.user.group.query
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.canvas.state.query
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.canvas.state.query, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.canvas.state.query
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:184 [z7wx0g93sdr7] DEBUG - not
> asking for -1 credits on acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.afterAcquired:98 [z7wx0g93sdr7] DEBUG -
> AfterAcquire 505 credits on address acf62840-864a-4eb4-a902-4861db94eb93,
> pendingCredits=65031
>
> 2024-01-10T19:52:34.824 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d43c72b6, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,FE9-P1M06_4-hpSI_YpzPbXBXnJJatopeQD2ZYxu3Pf9NWzlCotC1BAbSw-CkPVwd4p-ZF9OLD5UXIFMFY3bZIwkHhzIo32b1m0CpruDodE-_f-W&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,dWlUyTFJQxRIqOG8poa5pxge5mYVJ48dUXfACv6LUtKsZzFSVGKyZagOScZPGCnB6cxsyzdobSyk8B6Bwd79504jcLncGDwqMeY-9K6dpOs,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,IJ2rIXxWuL4HV_0QDasN4Pt8dv2RRaruLAddl_V22qHM09tvT787Et3f81LH8GEe2qDoNwQVHNYo108yuNu3b1QmP0RABo4i8D9b8fEGVe2HYd9QgjXy&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ZNE9b4nkKfo7M-_v-eXIZ86i5mBDuVpXhcuhH7d4Z3TXAYIWBA7iz2F7WJKgkGmWz9vMjcTKd3WNKgAqq_OrNuwfMP9rqwnZZdfDvOywCznI2b71&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,jYeMQW0WCHRb7cuJ1CqLy_AuRfdEcEvEJyhW319lWJzbpHyQUcNdPzWURfHTEtR71d2r_2lEhQf1hKps5IbaEzZsNJIWqMVdW1EGF781kPIabPU,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,ud4sg1bGnGSsxR72jGNWKmRRRqC9RhLjc5Fa_mAH3joB4qQCFRRYETYRnfp_427BSgD3-id4mG56z7G-3qi9hl8xOglhqXamz4f_kpydTW4Drw,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,LKodAPH3Ka3eY-QiWDHN0Dy_mvuaRX03ueKTXS9etN2qHhzBavU5TvH8lfQjoDkin1OiPxB9M0-rKA8tOzft_WViz6TpZZ6NNkERWJmT8-UKzfsFuw,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,5ea91cHeAd9MIKFvy9sbzhVk4n_LaL3aA10YRDmCls04v12YLb2XMmXw6eBUcLVuIE_Z644AmEt_e0ljvQnrS_I6HavZOLh4zjfiHWUvesg,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,3EuUPQYl08ZdWQB64BkecRgHHYvSChwe8vbP1kub9GpHNfB5OCg1ZRGw767893o5_ljNDyszaWPa4bDjjEuzEB5SNMICoB0-cPac73Tj-y5RWpDaKHCkuwVo2L4,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,jDru8ObAZFnt6UJuRwWa5fbG9hi1-FGT2jjgjYcb0j9MnKP4NE8WTbKBHSHyKvKnv60VBWqMlES8saZuop8zduWr_U-bg4uKdFl24xth0Wc08-CvUBpd&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,buy5utXjIbe-DcJ7PlEjzXqjMt8mDipzcDzczhLR-izL0Nh1zK5MFk42iry_zn2VPAZ7CZ8GCTQjVt68Y73BXII1KZvBi1ATYfTqOscCWbX0DnlP-bE,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,FCoOt1eEzJLmEhVMnIIHAKa-kTFceMfQD6Kcwc4ZSxQI7u6ROHuAf16piW1PXfz5fwuSX1g87RD5aUJh-U4htAPPBkPWT1OqrJG8TvqS_LoMCTdPQPVU&typo=1&ancr_add=1>
> )
>
> at
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,uJwPcj3QaOxYK9Va_aF3DQjRZaOd6gkORxphEXx8crbnrWTQpbS4NUvdGwXFXV2QxdoFEiSVdKtLtallasFVmS_ThZCt_nFVtdxSiOhZB0BxRxxTXdy8ETkT0A,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,RozNWdaPDnLkxzHk0PIIu7yOI0mq7J36SJtiYqHJSj83AaF6otFCOMT6S0SrKP0i5_HdKBbJFOfyhgDON-XzQcdSOvC2Z4N90Agiv9ZXimQSY87tDA,,&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,cU6yMpGHxe1nbp7Kzrg6EeEWtSIupu9-5fpY1zduuLu7sLJpcOXhuMMDMhuVH92Zbgrs9uoRcfmFq4l74iTpclqRJEOpXUr_QjhNgLoE&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3xUOFzb6MCi6u1j2LRQL6CE5BWbu5eM6Ir7EsfF2oCaYUBtqnA1r3VDYEmgBiVBb7QypDxbOzY9knH1MLZZ3oikFpY_p69_553LHOTbV7g,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,flQWOTe8aKezv5fxKav9FPsBjUaARCUkJOdOEGZrl7wCvt7GugyNdNvZClWXHAEd9rpj_ZWpZp3Cafpzk_fnpUQ4T8nZmy15zot9RNiXjba71GXeOkk2&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,9xoBaA3kGi2K8zQ8BuNLDq0dYmsnE1nuJw9sP_cDKKpUiUl1eHBvRBJF127g2yN9LNqLqZkyPVSq8YusAWP6k6MCI4_4n846oL1LY84Tow2q00oOukNL4Q,,&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,i6pIMkwS1l8fvBdFueQgC2fd0K8urWRRwGCKI6oTBX3N5vAYCxnNUJBz41YEp0I3NQ4S4ZHQ194n_z9O4VXiQ_Jz-cSzO5g2wCy4Jjpoams,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,OqfGI8b5WUrx4fzs6Vhc7lLcb55MI-F2H4j9ic-9Fbs1_FhBfTLzdobmjbVxUHuNV9kcphYOQdlHoTf8bN7hu4yIeKsdhwLW0rasAMuEm4mWhNV8t2c,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,j1hybmOnVH0IFEuunwALaXX1VhMjfTNyNAVYi40bd7VIdfk_jOox9LkDhVy_uodaH4IoC4x4qyX6rCNeuIoEnqF4PjDggaSOdEh6HJe1idmx4NGO8AI,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,ml--e7XB5_TgtwWFh6lw23pXHYNiKpjFGKoUKmwrOwHyS3lnFQ78O6WUjq-OM1ITxXTrgD6P5K2E89TnqOgGcaOyn6aLaVIcvPhARMyQEqlFDI2mcnYwIfwADKB-&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,QIlD68RsucjML4is9S32DJRE9Z3dapep-RlKt-pyrEuRTIt_G8nWBtRgb6Q8IMi4IJrlS-LAnOypotHiXNa3NnJWi4t7uDjt4su7nlkZ_w,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,olHwwJjC2TUJmGxt3e1EpgRMfVMAAdmudKrqZ5ZByilJc51VAK_ZZHGL1mLqUqv59mtQqrIVOT0FHxcuVGnfmTsHrtP_Lfh6vEjgZmLr0Aa8EqAKSU0BDQ,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,oMqaoHGrFSMkCOPl4uTKGzTSGTpfT8r39rhdKWVa_mFBDphxOPM_6yaRZJfLYNMa0AyN32-362kp_uiD-mRcEgDlEH0vkXHwcnpH_v5piaep5XwKlX3fJLhn&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,3crdMkhGhlkrJXp-HsjZIEJOjIoajfvvI_g-uTOHgEc26-rm5spEwaDR4tTrIOKeN5HSQz3HfKcnanjUMfuquaMfvk7M7K2AtlUVxZdR9Q,,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,3aPmcLDsyuHhnVLEv2uuGHy27dRF0jG1E2N7oB2o3YLd64XVZE-Wu8S5GjeDGqfzFEPFWOKwMR990Sz5Rl_09u_nU65WDNi2AS9_hDvMNMsxtfkYZuZNMQ,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,Qv7RvLuSjcsV2_72qHHLM6lmOI05RppEGGyR8280SVuKWQgOAl-EeFo8ZDeB_9Hw2f5ZgHbKDJbuB11rT-FBX6wf71ASJsIGNZYXzHn1HnNPy3pXaBaEgvrJvA,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,KLgaXgh4AHKpDe86W-r95VdTmG5KxMAi8sJty_SCxIHdKsN17C5SgWizyFoSLnNCx82oAkl1IQ7G3WdltO092d1YUzVphf699iq64LfNMcdXCeZV&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,7hPEKeOrrW0p5qUGMytIu7avFNFhxNhXFzKgo6a-hDqm4ky01dphgxAxTpF5DmP3oUrVV1uWYwMbJuUFv1IoX_o9-H7Y6xXWdj067c1A&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,6t2DJJUOLlA8obI6HZE4TmU6f_q6BKzPgJBJoo3fUc8j9wMievz5OvhrLMW74jh0Dtb_PEqVIlw_zez9B1EfFhl9WokvVE9IvfQ9vscZDtW4&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,dU7lk32qJcXeW-cUtM1E65BB47JiLhSgJYTqhLIX373Pj0miueYi3kN7MYtAU5zT39qOtzzAiHcBVYWsGDB3N8R31seI_xbm-rQzfd0jzb6V1rSLy1kotiI,&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,lKCCRLBtMK05PYln4EyoJ9wkUFDWR-bNCq-w2CS3bTb9B4_6-dwwglQPojSGYr1CeGxZdR-SeX4AkZ9wUq0gK-sVV1TN2CgVQFfAev2-3EQ,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,BtR0iFYtXaxmAlygBDvC74BLAVELzJivsFTs_YS41zS_XagPPAr0UUKW1lNrafWE3WxaZcW_S5WbNn583_uHQ7aLAA9vskUJfxaMwUrM_DxAOd2N&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,JM5uHqjx69DeHPuH1fzlCWUlAJt1HxJ7SMAoo16B-QlHBBzRndsmUvfqA_ZY1S6EPFV3mEGZBPAWdJteWlF9vXQYKEvGK1kyswVvG_maoCFPXHC3niDtcVnH4g,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,nW_o_ayhGlEOytZThvhNsFjqKDq7oEaYAEjw_jVKEHAnXglYBlSGSA3yj7wyG-2DXRXiRZSD11UwsKx7fSGPuHkUUSoULIYqacVjSrfmUuEfG_A,&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:34.824 [Thread-19] JmsStaticConnectionPool.onException:78
> [] ERROR - Receive error occurred. javax.jms.JMSException:
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a714&c=E,1,J-35FPJpVlAlzxdK4DRXRFHe8NLDGoFyrF5bd0TJMALAxu6IPXf7R38xF3V3S_obALUyJZio7n8nSS11ImujLD9S8AoKJWyIuS1mhUUsFA,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a714&c=E,1,3FhMyZ8H2FihvFzilmw5MqmH_tkqp5vHbCOJsN5x6sQNSLyFELuJt_Ej2M_24CH42Xx0iQesALGDJK8HHdR_PDaLploHa8R84-uwfftA3DUM&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a735&c=E,1,6plQ94hoZFwDxCVEMlguzKCU53o7QqHp52VpGSjXXTWicH1j4n6pp_oAxzWcmYFeQfPGabu5xKNPXO5t1rOzVCkEJL02B0Z0C4VpE1nN_Q-s&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a735&c=E,1,JvpZWdXDc673FDsIBWz8WgGu29y6j8H0zAczvf_UXYyUAzEMKtvXJ3tMkleaFsrHU-QdlAUi0bWddjnikuurWEEMfZs0jIDWZ3jnrx2__Vbg93yZ8oo,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a868&c=E,1,rouSt1axPk8dJSJf8RicvrYa-2QJv7ZdOGqbW06Dh2gGQv3VpFGKm4kveVLGLqGB3MY01qSV3dPGxTpcGGuAtlDEdii8a2mZYz-33WOWXnSkdPrCQbjy&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a868&c=E,1,AznnL1cSXDTbCj6iQ4ovEeP71-oopbCcH0laxV9dB-Y_kBuerNfeOAXRkRLybAqgqQBjJZKBNFEoveCJjTGncMCguHnY0hvsdZuKIHz6BEhvRX7N&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a856&c=E,1,53nm3gPRPTzcTWl5-TjBB6-0L9iUvYzlHKumiGfEb-1fbfZStPd4a3L0h41Kq1typVYdgSUQ7nT3w2U1ZuhBKEa3-WhSc85ptfbZ7I3VciWzT0Yaggw,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a856&c=E,1,vBewrTh4rfp_ezwl572jqeiy0VBShHwQJNRT99yv4B54VM5Qaf2hTHrOq33emXwk_WSFrcueJSCKI90AxcN4jUznfP1ZLzL1dJdUz01zK4vN&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a802&c=E,1,G_tte968_c7aOAQgyS3Ugbr0nl4R33yoPsC_W4flWhC16Em8YPUdOwKDzassaWqeuYxIOh5QXYq1l7ytew3cb9sGIgX_of0G7wkTgZJolWu0wekM1g,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a802&c=E,1,EY-ry2DzdJloXWtaemW-dWnRy1-rr2mQFFjoCbmWjX6xHCNZF95ldY2zUgfhOhSQFfzm7eYV0PdRUTOwJrx9jCD4U-B5ztFqWmvqEmJQuqdCyQgypA,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,o6cWiT7bpmgtzTzbYb3pAQFFxJ9bN5oZXHVo2EI_ad4rkkLBPQcK2Q7N1UbZ0ZmqCgG3shVKqOI8vF8UYSAVxzHE8bN4FoWJzNCFQuGHw0X11iUlFHdX&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,gHouFFMo7plejV1zi4ureD9QEDkW-WjvytRlH6DOyEt7oMO8ULkj6APov-gDhxtXAspTlk7qF6RYw-RrCqfSrXlcE0Cm45J61qJHt5Yc3zuBJJyEX8A,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,EYjW0HIGARwXmmVt6jB4dFiC-InWYcDZxydanyFK-waOk_olLAwd-yudL9xJHMzPqJMuKQqjns2IvmTemRxjKodxxofm_Zv4DqvDa5X5OxQzr-_L3Pz113KtNag,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,17MGSyeFWTwZKhb2DCqWeyFvdlinlKAunc4JSHJltjjiaCeT_1UjXFsTMtEiF_NYIOMI3brJbPkRW1QkJIhe_Br44jy1od8-BI3l37IfFJzZbcculrc,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,d4k7cBoCYIC9j51ETZxBDEayFc1056FwCQHT3d9loHqh-7uyMyVqmQtyevBar-ule7SfuAHPBx-chAkT7YfTODXaobYfOO_9b6if9LIg&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,F6NyfiAiJI9fxHnOQnnpO7dLyoFavqwnNivvLL6x2LO54G8SpNhXjlUN748SkNWwe_oCD9JhI7u_BOk1uNbwSTYV0lXRyewBrwvvysSEOSCggiFCFRgjtA,,&typo=1&ancr_add=1>
> )
>
> at
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,Esh9CwiY8Q8wRaXAM1ofCeHlDikAoWnYBv0KxS4ZX7xLt0h0R1rYpHtiLaELOsdwxhkfV9L6N7QEWOc4k-RiY6aOzby7BtUd83a0wTnmzJr5qrfSgGVEaF8,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,TnFdrKl6nCZv9PR6lSsRZ2YN0r8LqpGQMRo3oHlgyE7_AgUMzrW12R4HeB5JhX8Akv-1iKeVGrnTLV2WeAbBOCM-fukhfRPsdUgq5pwe0yU,&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,DVdY4t00amwsXk2vhaH6zLZaOeXBORNxqLIn2J7uyd6muQgSoF-SBrv0E5EGGpc4YB4mgXirpL7_C_uKsTlrYDo5DQH6IceC8gUL4vT5jErZTUFdrTHi&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,vAtPtyOuzaufOELKjMWnOBMrJEOShH0EzH9G3WL8gak4hWHmnVWPL1W7c_YnxRH2IerBQErg9VJApOiiX6TR6tcPau3xqRHrPyrsH70KHg2v&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,1QUkZh--OnzafOeQWcKcE3LYcxZBZTS7sFTwbWk984gjzF9pqi5lqW4RgsCfU_g1nLbrmd8LT8IEYDfHskjKl8sphe40gTARfNmI4gGx7w,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,pak9EX0xrMkRTN71voUljZIOmUvksqdCCqWdwh8fq2R4OZXUQFIHNH3P9j0t-wpQStfSs_ZsT0a9BZYopaoyMSnMkN3xqvXn8A9dTehv&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,geFSAtYIGv9JWgw2y9daWnn5EtrIqvUZaryRk8-l91NF5pD-L8bV4mxozd_mcICnm18YfXjygphpodR-DltMQ_vHwUd-4OWay36gD2Pwodoxtjs,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,acJEflBFObRkFNBskLYk7AARfS1_CzZH3yFGYB9r_Ype7n0DAWduZhGpmImVc9k1pTnmFoA8L8UukMVvWBinxujU6r75ipmzh4I_cNQ5Gzmd97g5SAU4dRsf3Hw,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,nUBUrDCvls3gKM5PlBD2cIXpf1W6rFXWSguqSV0S4dzfPj90PFVFAHWxCw4eO63hLAoTxi6uy8_2NIh0YHuWiBGYKicU_56rG8KAqSNsAWLRLDtAO0RxLw,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,aM5tVNMvOtLUxuoV8Dez1Y7fAS8KwME3peX3IqP_QPViIWgkZQahJW_1BMBQUo35u5h9hGJ3XSyDgkECwWOGDn6a9W-gctjh8WvUKx9yQ0sSLwheMcs5mhR2yVM,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,6p9TNE7Xs6QBgPslw7yBqXl57NwtLJGLN7k-pYgJu9zGoa-hmO07zvWfD_kO4CbadX3p99V2Y0lg94ChyckIxDQQqhJE1qJYz-J0cdZYYRpPQg,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,fUWcgLX3hupjGCjPYEByZ8xfzFx4iN-VnkLB9MarNu6OEOfrwnCkvJvtu6e-ZV-FREp0Ssisi_fezDu18eP9adS4mmcPPlJH69vnf0YDVg,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,uWfs3OljpnBLADqOUnftuzzQ12hzV-_CIwYB7OuBK9OgZgUZTvMdZpSPOqxe2vDLn7uXOWQCAeQAz5kh4iSGoPSrMeWO8GoHNI52KZEa&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,KuAjbB9H9ddyi6unfXrVdNXylpB7l6qLIUfJARScyUnc7iH1GtvO0s1MAtB9fwkdqXq8idOJYZs_rg0iU-sN4aV24ID0Os5tsyvB34ba3ve2D3diROqixzkQuw4,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,uCnHZgtUpZrhEpoalGTGPF9_Zo_vFtwZpK7i9OECfoEAgHQAk9HXrc4Ai9_T2uwdRSTm6kuInb_HD57GvWWLNrX3hqs8ERIp5QXo50ME2UPVvaYugac,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,EcYd8WT9GtMN9IXo58gsQpj5uNwcdCRtDfa0zG9t2jpvfdaSX5r9vLey2icJ-4wo3EyBMyUQzwyqYwC5BMfe4yHGjyo7h8jxmjcEKig4V0KyVa_SVq58NBT8&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,7xWJeT8e9at1yRzTwb5eaQ-EDFP7kbguZ1HlTyS15fjDzDNz2TZlYDBhylM75L9Y36HGRriKW9F69ImAAFS3e5X_53PDBRJOQPtATaWXMJlwz089&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,_a7OW7NorEoCm9eOnYjCi6TFl2__or_ickK-DwP4rrMmRX2Rj_TKaxpGjSAwCJZasllt1gefhShoob68MthMyp6bxv9N5fucG2HiF0g3yK4kqHTZjiFn7kxZ&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,_6j-wwzNJpJ6yfdA5D-GsS-QP4F7fbTyvCvRsTCwS_mB2qDx-lBWvBexxnGAMZnGwmMBeMncW4HkEczGZS6veAsx03mLcMheAU29hvOHDpRCJCAn6uwXUweChw,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,-ttWHQzQDZ94E-HqKtYTKV2nuApkTk60Zy25iWFIykqfNe0o2seEul4UE_gETZr08B40bJjmZKMHq_-R0M8bQ8Fbu8twSkWx9MPLFUhc&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,oHfUtnglQcVzm7c5ytqIWCqg3Ql1T66OiTaXoYJycG7aHXcRVA-XGunwLIzdVb0E3sYbVFE9yOTMe7CM9jDlIGgGtbkvhp2_FUk6hv74Kg,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,624T4PuCyLmsS-UG0IoXRx3MMz4DqnU_1BuWfOz973WSpqfLsFFDAICu75KCoCwAs9O8WgKTEcBVqRhhf1JUqI3Znxem3LsG7Ml0Td3DnJJUoGYR0BE,&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,nsf2ybbdjymP64mZ0JsCbzYoQjO_4UlQcKyh5v3NAla2uaTeh5sLx0V7CeR8YWgzcfu2SaTVEzv8aGCf5ha-xcrCH77k0UAnZgPtmcXMFImJxbtxC8ZJD8J55fHI&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7UL50c_ahB0IhSHbe9OlGABBKKE4OFyNM67SjmNrSo_LQhfViHAlLgvwdlRpbmDdymI5VaeOEklHjvlweLBz8Ske6wjc8J3CzM05VYenh6DzNiI,&typo=1&ancr_add=1>
> )
>
> Caused by: ActiveMQDisconnectedException[errorType=DISCONNECTED
> message=AMQ219015: The connection was disconnected because of server
> shutdown]
>
> ... 7 more
>
>
>
> 2024-01-10T19:52:34.828 [Thread-17 (ActiveMQ-client-global-threads)]
> ClientSessionImpl.acknowledge:1225 [] DEBUG - client ack messageID =
> 10054445
>
>
>
> 2024-01-10T19:52:34.862 [Thread-24 (ActiveMQ-client-global-threads)]
> JmsServerBase.logRequest:117 [] INFO - REQUEST IN:
>
>
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,01YVdf0IHIl4w2xIwUh5i7ToRiQfv6TxT-FRWSNW0n6e2aucHkWVyKxkYJTpG1jt7_EUCuQVS0NWGalDyf11AUIb3qVPj67IUtjJB484IdLKrHgbmSd-Q8Vg7AKZ&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Tuesday, January 9, 2024 1:57 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Both the "replicated-failback-static" and "replicated-failback" examples
> run without issue out of the box on my machine so I'm not sure what's going
> on with that. One thing that's odd is that out-of-the-box the line 122 [1]
> doesn't match the method call in your NPE. Perhaps you modified the example?
>
>
>
> Aside from that there were a handful of questions in my previous message
> you didn't address. I think the questions about thread dumps are the most
> pressing given that you indicated you're deleting the pod. My guess is that
> the client is getting hung in a network call that doesn't want to return
> due to the pod disappearing. A series of thread dumps during the minute
> delay would confirm or deny that.
>
>
>
>
>
> Justin
>
>
>
> [1]
> https://github.com/apache/activemq-artemis/blob/2.31.2/examples/features/ha/replicated-failback-static/src/main/java/org/apache/activemq/artemis/jms/example/ReplicatedFailbackStaticExample.java#L122
>
>
>
> On Tue, Jan 9, 2024 at 12:47 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hey Justing, thanks for the help.
>
>
>
> We are using version 2.31.2, which is pretty recent, if not the latest.
>
>
>
> With regards to the log traces, other AMQ threads show similar behavior,
> and our system isn’t doing much until AMQ-client stabilizes.
>
> I am running on a k3d cluster and the failover is triggered by me deleting
> the dm-activemq-live pod, which k8s then spins back up.
>
> Also, our slave config does NOT specify “allow-failback=true”, so the spin
> up of the new dm-activemq-live instance does not trigger a fail-back, which
> is intentional to eliminate failover-failback chaos in such a short time.
>
>
>
> I am trying to setup a simpler test case by using one of the examples,
> artemis/2.31.2/examples/features/ha/replicated-failback-static, however, I
> am getting this NPE on line 112:
>
> Caused by: java.lang.NullPointerException: Cannot invoke
> "javax.jms.TextMessage.getText()" because "message0" is null
>
>
>
> The NPE seems to indicate that, you cannot call consumer.receive() again
> even if NOT acknowledged, which seems to contradict the comments on the
> code. I have run into this same behavior in another example as well,
> artemis/2.31.2/examples/features/ha/replicated-failback.
>
>
>
> Can you please confirm the expected behavior of the example code in the
> 2.31.2 version? I can then modify the code accordingly. Or please point me
> to a working sample code.
>
>
>
> Thanks,
>
> Lino
>
>
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,lIOPl76F2R4EtenIAEJMeQ17kfmHQ6buC7G1W5TRefOcSfxvj_-3dLXV3MKDkCTkac_65Rw0SYmof66i91zGwgb_SS1nKaBpao_Lw7JNNk1mkE33&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Monday, January 8, 2024 9:20 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > Now, the backup process does start processing msgs right away...
>
>
>
> What evidence do you have of that?
>
>
>
> > ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute
> apart, before it stabilizes after the failover and is once again usable.
>
>
>
> The log snippets you provided show a reconnection attempt back to
> dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to
> dm-activemq-backup-svc:61617 both of which fail almost immediately. Then
> right after that another connection attempt is made to
> dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e.
> from 20:03:44.874 to 20:04:44.933). Are there any other logs during this
> minute? Did you happen to get any thread dumps during this time? If so,
> what did they show? If not, could you? What was happening with
> dm-activemq-live-svc during this time? Perhaps there was something weird
> going on with the network related to whatever you were doing with that
> machine to trigger failover.
>
>
>
> You might consider simplifying your test-case and then adding complexity
> bit by bit to see what triggers the difference in behavior. For example,
> run your test with the client and both brokers on the *same machine*. If
> that works then move just the client to a remote machine. If that works
> then test the client just reconnecting to the primary broker upon restart
> (i.e. no failover). Then finally put all three entities on separate
> machines and test failover. Hopefully such an exercise will give you a
> better understanding of where the problem might lie.
>
>
>
> Lastly, please indicate what version of ActiveMQ Artemis you're using.
>
>
>
>
>
> Justin
>
>
>
> On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hi Justin,
>
> I’m working with John Lilley on getting an ActiveMQ HA configuration
> operational. Our current HA config is replication with master and slave
> instances and with static connections, and it seems to be working, except
> that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect
> itself after a failover from the live to the backup instance. Now, the
> backup process does start processing msgs right away, but the
> ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart,
> before it stabilizes after the failover and is once again usable. I have
> included the relevant log traces from the Thread-0 ActiveMQ-client showing
> the Connection failure detection all the way through when the client is
> usable again, over 1 minute later. I am hoping that, based on the trace,
> you can find something that, perhaps, we should be doing differently in the
> client side?
>
>
>
> Here is our Client connection url:
>
> (tcp://dm-activemq-live-svc:
> 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
>
>
>
> Should we also be including the corresponding connector names?
>
>
>
> In an earlier email with John Lilley you recommended 2 brokers using
> “shared storage” instead of “replication”, which I will try next, however,
> since this issue seems to be on the client-side, I want to make sure our
> client-side is setup correctly.
>
>
>
> Thanks in advance for your help.
>
> Lino
>
>
>
> Thread-0 log traces:
>
> 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-live-svc/10.43.197.248:61616 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=a8a4a77c, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> ---
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and currentConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.197.248:61616
>
>
>
> 2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.11.76:61617
>
>
>
> 2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> ---
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and currentConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.197.248:61616
>
> 2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = ced14fbc
>
>
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.11.76:61617
>
>
>
> 2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
>
>
> 2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 10, performing as a failover operation now and recreating
> objects
>
>
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,vvTQQyz_c1orZi3hJuq2ry1TKwMxLwK-5lFF_d5xs59Uqkggbs33ePLHKsLtFDFcSC0poz3LyKicF4cLJqyq7vAKuqJFApvRj_dDFYycgco,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,hqiYs6vnJzfq-X42G9W9ohJtYgn9LRPR-82RHxMyPnKUD5xBwCbo8W6b-PegfmAGwvEENlDO0k2IkoGFFegGxR-H6J72UHHlpEFbGtJzhCF3sucQ&typo=1&ancr_add=1>
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536
> credits on address https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,W47wSuoEjpx7WYHr-JzlVVX-Ynfz-605gpt-36aXWsY6a829my5okqJZIqnhM9xZ55Qg3K1p8v_plEA8dwM51sR9xD7gamJmHMziyTUbhngPiHtc&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,KX9atZvl9XxpGPWobW9hXsmaLrVekfw2221zvD0PJ9Ik3fEQVyHKcZZCe7MQRciamxD-T-WDEk5Xa7_cAt0PSHwVW3jG7zzhq8XH9XIRC8EXmg,,&typo=1&ancr_add=1>,
> needed = 65536, arriving = 0
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536
> credits on address https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,BWFGc-lOAMobDFV3VeJDY2Kuijq3pFDr_VCikUkusDdeYbb0Vb0sQyfeLak8kHOd64VVSCsk-i8q-_Ow4H077PR_JbUcjLwm7c2eQxbcvpB7_i8OP58ULr4,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,ZOCwWoQv34cYLuw9hZvJzBHfSrX6oQeQ_3-SetFMJa1hRC6lXgiKsAElOhfZ_vfnBhRGRJPZjSOqHIbwWO2ltSYYY7ElQjyItp0uDo79_jaDkw,,&typo=1&ancr_add=1>
>
> 2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=a8a4a77c, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,Sofx2L-Lek25HDuPLaBmuRbxNTjT15ZAkPaa3bt8KdsN8nfIAnfK_-i_LGVt7VIPBJAfTKCSvKdhtqF5fwMcZeiqsq_iCMdD1bBW62tm21E,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,obbNTaFqn6mglgyZXy1S-0nBLNVtEYlkZBwAEKSGDGN0RuqlR_-7fIQPW4ELNqVFJdenAcKh65C3yQ2b_Lef7W01Tst6ZTrmI66KfHLae572jXv-HF3O&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,AdTG_wVqCegqwIBWNNy815gGCWjYHluAdoqWg___kmDOihDqWodf-WPqu_G60ChwqSuOGHLMOmsRc1Gx1sxmV2odmtdcyxMRdFwNaw7sC18VSdo,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ytVVTXzfq-J1NXl2ct7T_47F5EbfZNQTXP8dlqw720EiCe32c6CLwlpk0-v4MzUD61FErq0nkRTalFpcTtIN8TDeBvz70rzb__TyeIDm48ZKfFU28qQ,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,pMAiGcinEIywwqtI-_h5UmfhiVIsj_wZxAwWzZo6qzUyUe39gOo5cuTJkTLvLOnz5erj7M-o-QXlxwx3N0QsIOrwhB8B_GcuBeAG_CvIIxs,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,KQYUuEgOaR6V8a_ZC_xBpKZmDneB4LTL1cQzn2Gqtla8ntHJ1fgRs9KelLgvbMd9zvBTg8maXUerYZYQZa-DlfK7fon0JZ0O74PNJ1SOCHqIPN4fZQ6Xv731OGOD&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,5BrDFP93zUeig38-IRGrnzjTp7JfWTb1WzEc72yNCIoRDXcwnJ_1esXJshYjOciZ6vjMykJkBZ_6HqsaGLoyPvPLmtDnSuNDgOa6TcAJ&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,OSVh6NSORxz7qZ6eIuUDgWskoYfyXT4NNdjJYGMbH7HeWEUiymys2DnptjYKdmQfqjfHmI-egRZ31yWwd9MKzEd-gbbsuTUsqn_46niKaw,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,McAPJp1t6gnApibCFe32C-8Cr9g_7ABZfKNnngO8C7OfjTCwbmlQoI-MGveBmExlr2kJuXuk3e7YciVvm-O_CJ_VqB6W638bjVcnc0rTsdoZ1qV6-i09SBemJYQ,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,DM2boEBGg44zzcd0m37ITkOnfDrkqLE8ZSFGJpEI4AGo2GyOI2yKX08AEkKHxw_UnDMF1vccYEvESFDXjXBBjjIGO5TEsMLWZHbJF3AOIEQ73uc,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,_8_xgPRaoPIbRD0yVdFOK2vlKMKwV-y_AyZkl51NnG9Bken-pX8fV_6ZnBLL_1M2NNT0EM1uFQ9SMYHiFv6Z7wYtXfzmARfem638au-oY_Tu7_4,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,wipgGeA7UVZIy1uuqvuxJEx4BA8TvYfSYmO617IgrvnOo-z7sn1vUjXFqIEU9M1HmedBVoHxB_LpvuEZ9p_biioPM8V-V9eS8i8oST46GvBhZZIc&typo=1&ancr_add=1>
> )
>
> at
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,m1mdePfZWYA-kcbFesMq8mnF6E-jEN-12F4t_ogHcJ4G_XcN6Dego1WHs9hZUnGwqRP0K9OW0EyJdfaKVUHimTHN_v3oiAjtz_1m4D92uUWM6XqiTq1AuMc,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,0FNTjscT_d2CNIid0J8d_x7C7Peb6_neE-a_MZeKqkM1o_sPGujvpkXuiLv6kms9wSFyYqSux86C1MX2R8WKM8IZBsjKt7sCs03U2lRYlVbw2eTDg0ihSEV1nA,,&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,_rouNwhPTTi2XkDcyj5UbDpjM5Xb5S4o5EbjNaUZZ5kVwkN5fUrTM6G9u83Q8t5MY2kKWp6eDBHyQ2XboNiD_3iYIZEN3rbdknbKtd2zGdiq&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3yKPqxpIi9n43QJSCQiysKGlEYttZ5Dmuirc7jUAWNz6orLfIUCYwdDEJ3eU0QiHywsNAJdWqrvnx5rZK6-8TTJb6o8tr1WSvSL7omLF&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,5f8NdiJ-aPnMigLbJtzN83B8k94qmJQq43nhmbgzr_xv-_XJ5Nf2ZDVZok6RUKOHjdb0vuJnzm-n64Eq2fnio3oG5I_tuNWTRbUXKAhU&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,6B4qLrj7q9DzDb0Trp9xgst7cOBhbD4hcEcGyPFul5E_mh3_oGmColB9YQkQTB1gJ6x5Mcm0VwvBLddpDpfFkcZwlzb4YNR9Ym5fcDclFW97pIrQ9oG4&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KYDC8kncdxnp7pATIxAxUBktC_iMOSb28z6Hw5CXdbb1busOZDrmEwOd0gz90nh_YCO_IYopJ2FTRxz4j7vr_0mRHXdlarXEc55kzAG2IUs,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KqR93A_gEw1FpU7__PsYjSkOC5LH0zFTSKu5K9YlCdVAz3h_c9EkBndO7aliE4RSbK_qslEZp0b-YzJfCOxmB22jAfplsYxguMl1qjC0rWpNjfr9_fh4RjNL3AOK&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,tiV6Vx6I0Xik4mg-tDUPMR48Vm95Asg25jpuyGMrlkTJBhdRMLVSEkT2KzpQ8s-Lxn5qQVtoTTc9EvJ1VH_fdwyMxShZj95-nohpqY0pNeoE7yE,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,mhChlKkihYjDw_Kk5SL1d_4eb7zhBlSzri_nwjOUsrs6MAvEfDAAjia07yTMUU6CQqc0WbDS2Fn8HJJI0UwJh1N7SKiz9dsP1gJLmgY66huLhXZE9c-3ECGXyIWQ&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,MtqIZFsxgj-Lx2vRvGSw7UH_Pzh2O5B_BYXHM7C3dLm4POkmZ79K15Xhs-ektW80o8GPKQrwPAdQYjjf_jDZ5b5nKqbmVeMU-qBxMZOF1Q5zPvBbgZNlZlU,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,nn8IDmg6XQJLLXzH7kunXDNZa_3A7fUlnHdRCLoha7gQWJO1QeY32duGFAQMYaFHsfIbHbv142mei8HfCqRwrpuUfikYxLkFoXaD3pORZb9roBJs0H8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,01Ld9YJlMjKksCSI3GimEsT9Tmtan8dI9-JcdLHZDdPqMABb2KMaE-oCZqJaiN9ZFK5rw0M8w2_bT7wc47JrPyfbbthma2evZrLIzSukQMFLpguO&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,DPLEvt-MPmmzv78LEkeXtYHM2q37m12TxX-nqjuIJurQdeXydsSakco9t_O7jgBbQ6y115xxdDb8IonFZbM86zgRDLcvUmeOSJuDmZybD_6JaTX0zpbQ&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,50rc-e6y2y_kWxF_bd6gVYJO_verct8DLqZaXGoNGN1m0VCTQyWRCDwOfScwBpKthaOqrPx6sg609tqQrG5vOR8VFQAoM-D0AjTKQXpwhx82DYiN5g,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,1vNFV-mDaLK92lJo0Mq6jvkCDzuwrSHT4lp1nB6KTzOCSO6comH5xNpEcvtZ_9pc0cQCD2kgDfB5xdxg0tAX2dF5oG5DHY6kxWXOBLnuAwiz1BkEFhw,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,SsQ3UwONvyPr9WW_l2zBPL-OwmRhL5K0tqgLY0-1J4ukt-wGoS0RPBCydXdNP8Y9HLJor2jcwQNr7eYoH6eyHr2nST0zn6kL1dCj0-XJ47ODQ1CK&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,aLydz0SGsclotqCro2upcTg44B_Ij7ckjApLu4B5JUwZKKdMo5z3RyUR4ikHFY8L3t_yB87Zp5atGRwoAPA1L9Mc9c681UjFrG16n-udVjeHtxQNxx8V-Nug&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,ceb67pKSJWHnVyXDhowqaqnw1emC2o1icbPvFlP1Ez2J9ErXw3aEcKlga83KIE47pZ6TOajqFHmkuHUTaNYJ98IMeYib2OlDC75rPTIM7IVtnExeYQb0CdEI9w,,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1mZX-u-XHEZ_e4Ut7YETRvWm7U_qkC-q9QaPeHah4K4yNl5foyM1VSExCtYiisknkelDj6_K2MUHGc65Kd-6-JG_TXLby0OPWodJVjX3BtKaDNBUzusgpzT6ajL7&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,DWbPBKB8qulo4SwULw5cFHdoemDsh1RZ6tdcphl6kJHY1lUU9R04kGYrhelrokAOssX6oqNOqq1H3Nx5WxGfsV89pNY82qtaQbNQC6lCQ2P9ZwHUzEA,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-nbEPDxI7J-DFvbJfVVt2Cs1Vi_hN5bw-8vQF3SlM1nthw3VbJLWmS1h73cupUxqBoHCImTzDqI1m5gutx3ZriLpLpU7IqutHCqZuT6CTVWACPtj&typo=1&ancr_add=1>
> (https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,KyBPFkXAqcb0BvrWcVSxGnvSM_xbYzAcZQZEAaBC90eFhNGTrXQWNgyCh25vjEUGfJR5fpaEOBpqvH2Jryu_FDRi9SNh3gLyraCHFr8UH2b77vuH0_JCHliGFLU,&typo=1
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,-lUcdhDzaHWdAWLCc0IxWCFkKtKHxoEd64GWhzXAbeuG_Zm9mvn624BY6uS8MgQn-s2TMF8PbEPYVbOOy6BJuI8ByeiuAF9Y8f2I5wib&typo=1&ancr_add=1>
> )
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,T2UV-1dji1RODRcU4Ff4j_IzfTH2n47BatzyLZcEOU8VROFr3GS8re7CjDcnnHCSaPEtR-35J3z30dnYcz5K66rVsMVxgDiyh0FvEDKnN67naHjYbn0NpzAc&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
John Lilley
Data Management Chief Architect
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Recommendations for HA/shared-store using Azure FileShare for the PV or performant alternatives
Posted by Arthur Naseef <ar...@amlinv.com>.
I haven't personally worked with Azure's solutions, so just adding some
background here that may help.
NFS performance can vary significantly. For example, initial
implementations in another major cloud provider were very slow - but after
significant effort from the could-provider's NFS folks, it has improved
significantly.
There are many factors that come into play for AMQ + NFS. Typically the
biggest concern is the "synchronous write" latency - because of messaging
guarantees provided by the broker for persistent messaging, the broker has
to wait for persistence to "report back" that writes have completed before
message production can complete. In the broker, this is implemented via a
synchronous write to disk.
Ultimately, measuring the NFS timing and working with NFS experts is
valuable. As a baseline, testing with different types of disks can help to
narrow down that the performance problem is related to the type of disk.
Keep in mind, I can't tell you whether the Azure NFS solution is causing
the performance bottleneck, but I can say it would not be a surprise.
A simple "micro-benchmark" program that measures synchronous write
performance could be a quick way to narrow that down.
BTW, have you considered the "inbox pattern" which pairs a DB with
messaging? Any place where messaging reliability is a major concern, that
pattern works well as it uses each technology (DB, messaging) in the way it
is best suited.
Hope this helps.
Art
On Mon, Jan 15, 2024 at 3:27 PM Vilius Šumskas <vi...@rivile.lt>
wrote:
> Hi,
>
>
>
> just thought to give my point of view on this.
>
>
>
> Check if Azure NetApp NFS volume would provide better results for you.
> This is what we are using for our Artemis HA shared storage cluster on
> Google Cloud. It is more expensive but in our tests provided much better
> performance comparing to anything native Azure or GCP file stores give.
>
>
>
> Also, I don’t want to encourage to drop K8s here, but to tell you the
> truth we abolished running Artemis on K8s all together. For our use case it
> was just too much work for too little gain. We run native Linux VMs with
> NetApp NFS attached storage. For the clients cluster is accessible by DNS
> directly without any additional load balancers. This way we left failover
> flow purely for the clients and the internal mechanics of the broker
> without the need to tackle K8s health checks, timeouts, graceful shutdowns
> and similar issues.
>
>
>
> --
>
> Best Regards,
>
>
>
> Vilius Šumskas
>
> Rivile
>
> IT manager
>
>
>
> *From:* Lino Pereira <li...@redpointglobal.com.INVALID>
> *Sent:* Monday, January 15, 2024 10:10 PM
> *To:* users@activemq.apache.org
> *Subject:* Recommendations for HA/shared-store using Azure FileShare for
> the PV or performant alternatives
>
>
>
> Hi Justin,
>
>
>
> Upon testing our “shared-store” solution in our target Azure K8S cluster,
> we ran into performance issues, seemingly due to the use of an Azure
> FileShare for the shared-store PV. We are currently using "Azure
> FileShare NFS on Linux" and NIO for the journaling mode.
>
>
>
> Azure FileShare info:
> https://learn.microsoft.com/en-us/azure/storage/files/files-nfs-protocol
>
>
>
> What are your recommendations on choosing the best type of Azure FileShare
> to achieve the required performance?
>
>
>
> Should we look at other journaling modes as a solution here?
>
>
>
> Or, is going back to “replication”, with 2 backup instances to avoid
> split-brain, the best solution? Note that we do not need to persist
> messages for any longer than a couple of minutes.
>
>
>
> Thanks,
>
> Lino
>
>
>
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Lino Pereira <li...@redpointglobal.com.INVALID>
> *Sent:* Thursday, January 11, 2024 4:11 PM
> *To:* users@activemq.apache.org
> *Subject:* RE: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Hey Justin,
>
>
>
> So I am trying to verify our AMQ docker images in the local environment,
> basically our system runs as simple processes on my local Linux system, but
> the AMQ instances run in docker containers. Now, this works fine for
> standalone, and for the HA case, the “master” instance works fine with, or
> without, the “backup” instance running. However, the “backup” instance
> doesn’t seem to ever start up fully, that is, it is missing the AMQ221080,
> AMQ221003, AMQ221007 and AMQ221001 log lines, shown below from the
> “master” instance. As a result, it never announces itself in “Backup” mode
> as seen when run from k8s. I am using the same broker configs that run fine
> when AMQ instances are run natively (no docker).
>
>
>
> Also, when I stop the “master” instance, the AMQ-client reports these
> exceptions and never connects. Any ideas on why this may be happening for
> the backup, which uses the same exact docker image, just with a different
> broker.xml?
>
> Is there some fundamental thing that I am missing when running 2 AMQ
> instances from docker containers?
>
> My goal is to validate that our AMQ docker images are functional outside
> of k8s.
>
>
>
> With regards, to getting you the thread dumps, next on the chopping block,
> can I just attach them to the email, or is there an alternate path to get
> those to you? Apologize for the dumb questions, but I am new at this.
>
>
>
> Thanks.
>
> Lino
>
>
>
> AMQ-client exceptions after killing the “master” instance:
>
> 2024-01-11T15:47:15.322 [Thread-0 (ActiveMQ-client-global-threads)]
> client.errorCreatingNettyConnection:741 [] ERROR - AMQ214016: Failed to
> create netty connection java.lang.IllegalStateException: No
> ActiveMQChannelHandler has been found while connecting to localhost/
> 127.0.0.1:61617 from Channel with id = 0bec947c
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(
> NettyConnector.java:980
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a980&c=E,1,9sVwvAhg-dCUoWlCTkknJyr35Kb1lORNDVxR6-uqw3mazBhZ5pG19VN92yx8WmUo86gmEfiM-fyes3vKIRRqmUr7xGxTepKfWp8f8YCxIKDeU3MjhJIbTwA,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(
> NettyConnector.java:866
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a866&c=E,1,PDO7x8ScbX2lkRczVyc7Qx5qFYh8jYI_QZSxIyD15vNJbt-HUSyFIAsL54Aas_i1kj247doxU1D-TF1f6UNtgifP4lOW0jVh-Ca8bmu8X2BRDUZyZYrlZd4,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(
> NettyConnector.java:848
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a848&c=E,1,M5S848Ei6ypG34y6nHCsxBxodJIiZs0WdncK5b_Paenx_yRNXYZToN-BG2GJ4M8V03LsOGQzIKSbFW7uKF8az4WwG4Xrzk0hIvgvf05YYBR-KraWq2Quc6cO&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(
> ClientSessionFactoryImpl.java:1212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1212&c=E,1,Zu7gFKlm4HmSOURefrcnPu7_omfp3z6NkhUulfWOlvCON5qFl8o_m__R5jhaoNXCjQUWg3afJOkgibeky8zBPzqugdJrnx3mDcRvktMSa8mHPzzyhu9E&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(
> ClientSessionFactoryImpl.java:1333
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1333&c=E,1,QLuXIqCAvTMp9vzA3xlqCzu6uu_CAV6-6B3ZjST9QtAqqlHPPW031VktMnXrspcOD4IBb1JKvR2F8OzNZIi97M8NdBmUvk48uDHiFLvxVM4,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(
> ClientSessionFactoryImpl.java:1253
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1253&c=E,1,HGg4Q1mxe7PWvtXuKXcO1vTG9pGtPTivn_BSZ-cDFd_RB5T6KlxMhp2Z2iKrHRFFrcmAJyClu3xTgdygcYT1cRvfTIU0mnOfoNn73hij&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(
> ClientSessionFactoryImpl.java:1496
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1496&c=E,1,YDsqwgPbq8QPr_ODsKRUE5-FJdCxEgI2EdJdlE_ogGX_Dtkdqu0ZUW5FnNQHKMwQfJEjscTg0sXGDaY7D27_1M-BNdURAJybIJ0mM63IpSP_Kix0E7dJj0xAAA,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(
> ClientSessionFactoryImpl.java:1074
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1074&c=E,1,UaHGLwmlvYxEp5Zvya3OgHoGpFPLrtGO8817O18DQ_eCtU9hppVQz3ZSqMOZR4Ou0RyfDnJ25Dsho8qeOiDziqkIZ-8smasmw6AS-SC3C07T9A,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(
> ClientSessionFactoryImpl.java:959
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a959&c=E,1,S2vAOnIQrU8itB1DHvtgyBp5ozbblk23_CGr3q-ohZ6Nt35WRISFB1PbqeDDdjRyaj8N9tzZSNUm_TpeEwNQg143NiFeTG18rXYlnq4jakQe2_H0HkGTrg5Qow,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:697
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a697&c=E,1,HRBbQntC7regy9ruiCBdgTxHOjjHiqqEtdrKtyuFoRluNvypXTYG7MHAazXY4rr-7iE9ymvG-ccC_EuBMTBG5AjelMg7KfPvqiIaUkF1RfMIvoeiA8o,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,WwOwlRDdAEBeA1Gg0ArmZFo1it0EZxIt6RsmaWdNGx6wo_Lz4qKdMxIHy7U_9S5bgl-33cfGzNtlYG9YM9FZXucb7-SjGMsXWWRiyGWMdHRJ&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:559
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a559&c=E,1,Zc7KWsYoeLt5E_P6hgERE60yuujJHXiWpuT9jFjj4gC367dihQ6w7V22qGaDAek1JHPJHKEsYT6vqwQAPm9GSln6px2RaJkIttMBpwOF398rKvWxlzAOmI4,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionException(
> ClientSessionFactoryImpl.java:425
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a425&c=E,1,BFL6YN5Kxy06vezMnsiL2nC8cdKvlemlxhvuHQ-FuWkoOzxoYzQuft1CeyeiQmEo-L40YbRsfc6dzmYWbWpQQb_xVyaLDbql7YwiFlA60a1xj1PSDmU,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$
> 2.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f2.run&c=E,1,sMyuWkYLp3LX6ibrSEXRcAezwbBM7qAOluPp_GC1jVesMs6jxDzxY4FFM3VjeigvvheAyp3Q7Oe5ACoqft2B-abCTUa6WF1bt7Uh4xjE6hHhWxo,&typo=1&ancr_add=1>
> (NettyConnector.java:1252
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a1252&c=E,1,3JJSPZ_KDZLXcsGld2w64FmGt132PiKafjM_G1ZXb-ulOdS6Dwv6XUv8QHJqHN6CKkClQ7lUkNnakzibZdks8vAdD3gLbxCOyjzJGO6K745CLD2v2F7UK5zJSQGG&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,bpAKAylKI6Drp540buFl0pkNNpNnX4xnMjY3zWwpNyF_392duOqGQPjpdfCPGxrpBvc7_eIyfJpxl4WuufWnNYLFKdIg0Nu9WDi6STLxtiY0tUhT7eeO2iA,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,CqAtDgvRoDEnuFjXXv-RSPQ-NcQ5-m4IRpyQ9oiN0rhM40sqLFbhY9BMbc0HKKA9gNrRlaFd_DPaSB11LNOv4Rdahsd5fJLOqXHsZnvg3ShSJw,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,C5wuD-LRtJ7605p6qERurgxxNLnHBKbC4B7VE9lfDl7f_JsOdDSkeiyMkYACWnhSckLFmjUW2eFKZawLMNYORn4gTtULbgzlBoD8LWUyNto,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,rMwMgnKWBZ0Z_t5VCFvBOA7J5McS_4eMRlTXeB6c0yMri_CRxIy8PDWJRjrRtRMI1QpqaQtuBEcG4sCoeTP06BI4NmwaneyL2iZ_EcqSA0sDvg,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,AmU9mOKxf3Tw98cD6iOKGEUsELr-mALW0vP5N2eC_TxCQIG4d3rxa4xf_W2vXyThIPw3Zmp6MJGpD2c95V3s3JefMOVZJGioR7UrvoKjJHS1Qr22gQ,,&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,uKLItaPZw48N90kGuOJ2giezmgZl49ttEaMZ2H4_Lr1mqtycK7JXSo7FWi90G25OhrIF9k-OxzV8LaDSYfMSQPd4uEksOAIsnz2-SSjTNFctPDccNQ,,&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,gIn498Y42-KQL16Kw7akU8ywkXBbJPMfxg7-Ivrxphmo6NJdJqsbcSdPEAdkwVYS3FgjewkEVfwxLXUtcBsU4QHUbsEaVo5060hl3qEBmZeRmUBspRNTPB0,&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,KRj2MYUCdG64pKABEvud93N21Qgh8ET58T7SDN4ZzK9ErKl92F_UyzJtq4y9eDIqY04euQg_cRGOTddiPcrePTpRRyy9U7AEk_WKzWUcU_VDwdQacLRCEHF4j6ag&typo=1&ancr_add=1>
> )
>
>
>
> -------------
>
> Log excerpt from AMQ docker “master” instance, showing lines missing from
> backup:
>
> 2024-01-11 22:38:54,844 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address DLQ supporting [ANYCAST]
>
> 2024-01-11 22:38:54,853 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
>
> 2024-01-11 22:38:54,905 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
>
> 2024-01-11 22:38:54,906 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
>
> 2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols
> [CORE,MQTT,AMQP,HORNETQ,STOMP,OPENWIRE]
>
> 2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server]
> AMQ221007: Server is now live
>
> 2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server]
> AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.31.2 [live,
> nodeID=320ff011-b0d2-11ee-a54c-0242ac110004]
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,WIQnILFOGT12V7T2V5JgdPTF8Y-yZZKUo4gSIgdk9Xb0RKhobbQh-PjrSOCrn9J79hRQkhxWG4u7HGnntjzO4n-TgjsgxAzGPARQB9OM7Zk,&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Wednesday, January 10, 2024 6:57 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > ...I’ll have to instrument the code for that, right? Or am I missing an
> easier way to generate those on-demand?
>
>
>
> I'm not clear on why you'd need to instrument the code to get thread
> dumps. You can generate thread dumps through a number of different
> commands, e.g.:
>
>
>
> - jstack
>
> - jcmd
>
> - kill -3
>
>
>
> You can execute these administratively or just set up a script that
> generates them every so often.
>
>
>
>
>
> Justin
>
>
>
> On Wed, Jan 10, 2024 at 7:43 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hey Justin,
>
> I did finally get a chance to run everything locally, and it all works
> smoothly with failovers in both directions and the AMQ-client recovers
> within 2-3 seconds and with a lot less log noise.
>
>
>
> I did not get the thread dumps yet because I’ll have to instrument the
> code for that, right? Or am I missing an easier way to generate those
> on-demand? This is happening in a k8s pod, so not easy to connect with
> jconsole.
>
>
>
> Thanks,
>
> Lino
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,P1-JTENl1pkpwfayJeVxTcbmyxoXqv5aWc8dgoVoQa2P9YJ6ITYanSwPPWI3OOvd-1ZuyvCfDdUgZSHNKSzXPNMT51ZJGff0MJI1JNYM4KnicxOXm0JmVYcKXz0s&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Wednesday, January 10, 2024 3:02 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > So our issue seems to be only for failover from live-to-backup,
> consistently.
>
>
>
> As far as I know the client doesn't make any distinction between these two
> cases so I'm guessing there's some kind of environmental difference that's
> making an impact.
>
>
>
> > Is there something that we are perhaps not doing on the client?
>
>
>
> I don't think so. The connection URL you provided previously looks good.
>
>
>
> > I will extend one of the example codes to use threads and sone of the
> AMQ-client wrapper code in an effort to isolate this behavior on a simpler
> code base, as you recommented.
>
>
>
> Before going down that road it might be better to inspect the ostensibly
> low-hanging fruit first. Did you happen to test everything locally? If so,
> what happened? Did you acquire any thread dumps during the delay as
> previously requested?
>
>
>
>
>
> Justin
>
>
>
> On Wed, Jan 10, 2024 at 3:04 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hey Justin,
>
>
>
> After starting fresh with version 2.31.2, the examples work out of the box
> for me as well.
>
> And, after ensuring that version 2.31.2 is running everywhere, client was
> still using 2.30.0, and refactoring to using the recommended shared-store
> HA, our AMQ-client is still behaving as before. That is, failover from
> live-to-backup, client recovery takes over 1 minute, and logs report what I
> posted earlier. However, failover from backup-to-live, client recovers
> within 5 seconds, which is the expected behavior, and our system is happy.
> Please see new log excerpts with more logging included for the 2 relevant
> threads.
>
>
>
> So our issue seems to be only for failover from live-to-backup,
> consistently. Is there something that we are perhaps not doing on the
> client?
>
>
>
> I will extend one of the example codes to use threads and sone of the
> AMQ-client wrapper code in an effort to isolate this behavior on a simpler
> code base, as you recommented.
>
>
>
> In the meantime, I thought I’d ping you in case you can think of something
> obvious that we might have missed, especially since we just moved from
> 2.30.0.
>
>
>
> Thanks again,
>
> Lino
>
>
>
> Our versions:
>
> ARTEMIS_VERSION=2.31.2
>
> JMX_VERSION=0.18.0
>
>
>
> Logs from failover from backup-to-live:
>
> 2024-01-10T19:52:29.588 [Thread-24 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-backup-svc/10.43.229.161:61617 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d7ff7885, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,O0LJ4VfRCemWEaMLULajHsgUo6YseAVDyBlrTm2G50vM20WhlTUV0ueY1DxMRAMStXAoc6wy5Xx3x1vi2XsnE5F7_iNbL-MZ03M4IS9G&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,VdcejKqF9B67eWPDv-PP_0gR6PEmUxFYHHtW8kyn3Gzg3q-HHzL-Lr7oztjj2LQumPJWO-1KF4OPavU8c3DW4zPgrUsidhnnCpoMUUn_&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,xKCDuAN84vJBjsCp75epCz7V1lR05TFhETdvpPcKntBTPnQk0zuuQSei7-ly1RPUUsHDP8ZAe-IevqwGtTGfEB_yNzw90QcOpEx5BaXOW6E,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,pq7BenEb7b9i-FYz1LnTy-gHYDdV-28FiSv3pkV_i-1TQZtRxakbT-rOKkgLgz2IDx875Dk5EE_OrwsUcew9TIsvrcZZZ9ZIlwboxMEq9xnR5odpQL8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,8mmrZ5alRiVlyEshqjlGpedpt7TuUVBJOy8DDm2qHLGoY6v3pR76llV59sMsYwy2v4rwLqiD9Rcf3Hg4QSCZo7rdO1rQgMNapCKji2jg7T4IJOk3OAeLnIs,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,RLTYzCoyknTJ1GB9szMq3Jg02QAnt7juedv1HCTYmmA4JU1DO6MlJhkvzOTBqHbeeOdKSvwkualBrhGt0fUk7Nl5YLzlgfepqTWcDiBmXQ-_xSMcAHPw&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,iYtEGEWoGf58Vq3CGAxBrH3iWZ1jWm1N1b3AOFPNlx0kCahcxHEwXT7ZAQ_h3ynXTeKJSkvmBiGUkbqE3bH7tGw-Gn8VV7R7LaCkxl9Dch5I4kQDfJDKSg,,&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,kH54RADAl8RcHDeCxJW9b-P4YJJFElj_Nbh6mHtmwlFdP2tKzzbK3wkAXg9tVqhIlsbd4gr2LJkn7DWZ4I-J8WVE-7UovVbVakuQTTxc2pbR9vuIDiEFztc,&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,V9-XUYgB7GoqPyGhEdE_CmtNood-yvXw6P9wRIqQOSYSWuTfiF1bhonQejBQfgJdHq8k2TsE1RAvcSOp_WzBE-ybl0l6rc5pIuMGdC1LUITpdk_Iuha7urSje4Y,&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,RNgnFH7dBkD27TX353Mc-kqneOLcqFXEgpGywxg8VduTTYNyYFt5VqkKfirozQw9V88WOI8XK4v9dvrtPIl-3u7cp673GvKxq3L-XgqypuDtSnkRIYnh813U&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-backup-svc/10.43.229.161:61617 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d43c72b6, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,L97BTn6AFLbUpYgHzAlQm9pDJ4Sn--TlGDpz05xOE3xd-qn9krfLJ6JzGTaiSpQwuVjuQ8KUPfUSW6N18pgw44-ay7eqYb2h0p3sxehZeXAC&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,PiedcDlDv-nxD-EF13k9j-sagD4clTOVvCs6xyDQvnrPzDUxc4xAbnHg0d7hnqeG4cVJZaO42LJvoNm59E3SmClgLAft-3N7F898HO6Obbkm&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,G5SZ7tGYpGBQzNRxdmIzsp4hWlUsmbY3_881k0N5lFoFBwuHt1dMCkKHI8wiZRGpS6qDY_UDBvDmKVQNRIHXIIyg5Df2nnWDCTLnbvWCO20fW0PC&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,MsagygznR7ie9qgJLlMogXq8ex-X4ST-fg_vbVbgcBanXur80DVykS1ftnzq0e4bWLE0k8EiNmlhlezemqTlBEncWMJz-AQAKdTxEk4WyVY,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,n7l9CUpZZbFH3ca7Iu2wxM3kNfged80mMRcu0Vn9wIoZz7aKzCP3c1OVR73iqF0oLbir7CjMHKAu3sfrR2cLs-EP7orqDWQKQApkzTZra9CJag,,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,PePvk640nUsvwJWNJY39J2R954rSgiFVPPSQLOsjYskMxtDnDVBuzQiZ8ut5SAPoqzV9uYW6GX3ekOAAoou9eu6VpHa_oFc2-7d27W00VBLU_o_Zv4wEUw,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,REmfZe55ytZRsqZvFkCNsYUcRO5Yd8PZwIlQfGystV0qUeF2mXSYaWKgleI098Ac40PND4ilF-ukqEWDY8N04AIRQgGgKGpG036xx7jNSc78YS6UbKWiMNUHfNc,&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,nnqfN76WKNs0ZYhVPzea17QO5aWkwmLqbfYrmQJ_qGQIgTzCL4PSXhjU9jy4mhbCUfl53cwF4ISYabr_zD1GUvCgv5m4dYW1_OROCBpe&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,vlJ5vYZAfrlVp9kL50a_jwWhqVW8-LMUgzeBb6mn-udcS_fMR1yld889bXgOXTMDiutFqKYX9XPh0RtxeXRLtnN31WqeOlH_CZv4_ZyhnOwXvm4J&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7Ba-VAkMm3x0E0f6F1f73TH_y9Ylo03hx9DEz4kqWzvrXy0gAv3D9zIM35eXIaB95UFxo72DkMpb0nKW_a-BIF89JobbRqjyVNLdC5K3L1EM520B&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:29.592 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:29.594 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:29.593 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:29.597 [Thread-6 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d4cf19d5
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.600 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 7b2392ac
>
> 2024-01-10T19:52:29.602 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.602 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.603 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = c1fb4c82
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.610 [Thread-8 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 06b7b3a3
>
> 2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:29.611 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 65680be4
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.624 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 97a109b7
>
> 2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.625 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.629 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 77b2de2f
>
> 2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
>
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:30.612 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = f92d7cd6
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.613 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 1d44895d
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.614 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = e8602acd
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:30.631 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 0a217bfd
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.632 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d2aa00d7
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.633 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 9a4f639e
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.634 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = ecdb1cbf
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:31.000 [Schedule-Runner]
> SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service
> as system
>
> 2024-01-10T19:52:32.001 [Schedule-Runner]
> SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service
> as system
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 2/200
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:32.116 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = a6b3d488
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.117 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = c0627cd9
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.118 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 71523eb3
>
> 2024-01-10T19:52:32.119 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.120 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:32.134 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 2/200
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:32.135 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 68efd0dc
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.137 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = daddef73
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.137 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 84813ce3
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.138 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 533990e9
>
> 2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
>
>
> 2024-01-10T19:52:34.370 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 3/200
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:34.371 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 1d8df71b
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:34.373 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = de20e193
>
> 2024-01-10T19:52:34.373 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 3/200
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:34.390 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:34.390 [Thread-10 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 4661fd5d
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:34.392 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d1c14a74
>
> 2024-01-10T19:52:34.392 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.404 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-10T19:52:34.405 [Thread-9 (ActiveMQ-client-netty-threads)]
> ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up
> backup config = TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc for live =
> TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.405 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 38, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.409 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-10T19:52:34.410 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 37, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.411 [Thread-24 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 10, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.411 [Thread-11 (ActiveMQ-client-netty-threads)]
> ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up
> backup config = TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc for live =
> TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.413 [Thread-24 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,S5999ok8Om6JjNfBLhC0R2hYdl3dGqPWsLpis0MTx38AhKXPNqMMlH41RrDz0UZsxZzSZbLvUEOYFEa7hwM4N7gAZfCS9udcqSmOwhLg&typo=1&ancr_add=1>
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,CF5ZPcR0aBH6uaRjLjSeWoBKLnRwUOOgvrD5MzWuerKGIs1FEa5vZF8rdF1kWSKbLwGQlm1IsOeyey0f1t7eiA1ApUgs1EDhL7ilSV0c6rFxEanucXI,&typo=1&ancr_add=1>,
> needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,nl9VlpvL5Lx7cxWUON3eMxxXV1my7fPq5WQLaWD1ozszhEPr1wVfqCwFilScj02MzRACloIhLZ8CgyMWoa_OqCu-Tf4folOdvKvtooRq&typo=1&ancr_add=1>
>
> 2024-01-10T19:52:34.415 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d7ff7885, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> ClientSessionFactoryImpl.java:410
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,awSP7U_0_MIm7Km9lwe59SrlpKsd9iUMA2y266CCeNZS6GJuEExdDYVS6EVMnlmPlVN0W1aqSx2Rss2Dm_YIM71s5j7OAUz6pAdGaTqnU_h2RyaoD6KyuoU,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> NettyConnector.java:835
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,8gAKfG0MZn0OMdhCHJ98evOezuGoCS0mNbGWplSwawxhF0ed3tPq9vKWoWUbn2xmoLJ34skkCTpI2tUuevSCeVNT00tNpFlOJlyAjGWu&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:666
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,tXm2hjddgUkJac0QpMg4nVt2gwg5e3ORdql8K6xsJlBZIxph-NuPt1Ks2iB0ZarzoFELPAuAerVMiQpEU3nkq1JiqLAju26hE6Y2c3MwmKRTj66O8tsfQ-WFSh4d&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,P0xsXGcezpL-KzlLSOt_rfr6b6RF51B2hthJzXMGD04QeyLyg6AnMvF4M8ZOEwF90ojRdT3CtOy0x9HBSpGK_mmF_6Zl7Muz2TKrZRLAxQRxhV5XnvYlwQ,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,qctWvKkM1LAyegiROEl3hsiEN3dPcBcH4jlkBConOP-G2cgGl6X6XG-kdeQuBBOC8BmtTf-VmlbpJTjC_wznn8I6g7pLOg1uof2v0j_r3yyK7U8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,KRtNFryEe6jhqdT3of2FZC9PF7zGlYTxJ3KPFwO6TViZkyaHiybwHP3JmOhQ4F7yZtorUVRe5phyCswDWLc8lIFBVowIJygjwxLmO-F3xJSonSQ,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,ltH-LxKKSiKev4cs4CgS53B1Sn1z2t9IkxY_vjCKeoXMalJfF0yTujWPogG1a9qkWV5DrAT_qCTk7MdQa3LpkAae5yHTe_19fREF16p6vMMJqGXqLjw,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,66enqLb5-TVz_cB0ZCkDW4SpX4c3v0jslINMsxhHSjXfBWrIFoM8PxfSS7_VWmc_MdfmjPHIKcLm4zfcCe4MggwEWLxB-RjiUMqnmc8ShDJLoit5&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,xM_aKOMjg39ORE7gTvaGKP72ytDNvt6tjanfNTzekduYVGBvn963oijBiSY1JOQeorfgcCf57lVZSYIQ20x8sOFAH5zNexuTI0rBtrkDrtcJpq-S&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,Jr0OyBaxbJX-r7nzGvHtUkNxDkYucNHeuCPW_Mqw5I4IS0dve_XCjayhv7uvFQTdinHTsHqf2ZtZAn_qz1ZnE04PEsE0KWGfHQUj6N5WIKcNhyQ,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,fTod1kMxyNrLe6wmpPVQB1vco-7IrNIs9Dx8h9kWYDCH7xYH8V5kj2iao_X4D62DX5S24Xi08LA-xy2zHwj-LIzSKOA38iBBAlHdMD7qu3lK&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,37VGZcLM1KBT3tttPbLrhK822id0s0HCTimMYb18Jk-8bCbWxvPl2N9_yFFLgXa5pC9DXbKX6c8DMHolUFme8KRWVQdBx_lbRwtpLIOp6O7uiuV0fttkzvI3jxdO&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,zIgfFXrrGxOB_lKTtscVc_jXQlq_pH7TMnaK0IkixM4MVqoSgc-Wp5VV-QGQWmUtW5dggcgtYpRa7OoRHCBvDew7fxYJq28eSZVguCiuBiH9DzdKTHAXC5o,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,b6YutPqRZaL3jtcBxfqj_SiU4M_x_Qx-GeSmBfL7c6CG24IUL7AdjJxpyosXZtCtKco4bSDRxOj7HJn5CWgbgbPzJ0TFuXzJjs7VRUKVhg,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,xdw7r1BqGJuJPUUyo2a9wj_y2KKTC-j8c0DAocpHKzpZZaVOgPcxdng0Xi_JMKvlGYmcUvlTfZMm2FYQ7pTrF20xWUreNaRQqtU6rlWB3kkuPmMNBALyTBs,&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,4iCAaiDF74c7cCKTdUIIlT6Qu6BmbQItK-gaAIrQtpNdie21ccXNS_dCydALepUtQNTgsVsVUs1NavHOugg0IlFK0CjTj8TKnThm55ffUeOx8zHc4oZUYpE_LRsz&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,qBGyQwtcrkAjVKJZ9cZzaW1ivhGQgGfhpcJUS_ww-vfcQVNfD9OjDwDX1XjpzJWYpo08MqSOrKLpgO8d72TBVtSGg2OQ4AmggCv3IUgKeA,,&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,Mm6fpsoFsFLUtanrkJ6cqlk2y1zCJbPDIfqxsd0pS_sfS05oTAHBNIXIUUd4iA8HJY0guikOejZmykugHqKeopxLvJE3UmpX9YX1Y3Gu2ToK-B8aRw,,&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:34.435 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 54, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.465 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 98, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.474 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 20, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.483 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 128, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.495 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 158, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.499 [Thread-4]
> AbstractProducerCreditsImpl.checkCredits:184 [] DEBUG - not asking for -1
> credits on activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,BYjREAStKM-0e-e5OOY86CxL1g9WrAP01m5ROYtsGCMHTh3wox28vU8_r7Y9e7QwRiDtJ065Xst_zM3u0oFx_K7Fr-ZkN1GBxPR6aWEKrm4nQA,,&typo=1&ancr_add=1>
>
> 2024-01-10T19:52:34.499 [Thread-4]
> AbstractProducerCreditsImpl.afterAcquired:98 [] DEBUG - AfterAcquire 333
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,r4o5lwz6oA1VbUpicHhTmEpC9U7BwWBMw8ZYMcrl4g-jVFykL22Cb2WJIZLOqv-hLbxRqgThYlmO40Jo84JsPHe-IYBjUU8WMQ-jy1lZqxq3YQk,&typo=1&ancr_add=1>,
> pendingCredits=65203
>
> 2024-01-10T19:52:34.506 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 106, performing as a failover operation now and recreating
> objects
>
>
>
> 2024-01-10T19:52:34.680 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 155, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.683 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 21, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.684 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 129, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.686 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 57, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.687 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 174, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.688 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 176, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.689 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 56, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.690 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 108, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.691 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 156, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 233, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 112, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.693 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 187, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.694 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 110, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.695 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 144, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.696 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 235, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.697 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 28, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 47, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 30, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.699 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 77, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.701 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 65, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.710 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 70, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 34, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 159, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.713 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 76, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.714 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 119, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.716 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 59, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.719 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 87, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.720 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 80, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 166, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 149, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.722 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 137, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.723 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 163, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.724 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 179, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.726 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 181, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 17, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 58, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.728 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 32, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 49, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 234, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.730 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 143, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.731 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 42, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 136, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 178, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.733 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 55, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.734 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 95, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.737 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 48, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 16, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 146, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.739 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 22, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.742 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 43, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.743 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 130, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.744 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 36, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.745 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 60, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.746 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 117, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.751 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 14, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.752 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 81, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.754 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 93, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 138, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 100, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.756 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 62, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.757 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 125, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.758 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 15, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 99, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 44, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.760 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 172, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.761 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 84, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 79, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 122, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.764 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 64, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.765 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 88, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.767 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 151, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.772 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.773 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 96, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.774 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 27, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.775 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 73, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.776 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 90, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.777 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 134, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.778 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 35, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.780 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 118, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 186, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 41, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.782 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 31, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 150, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 145, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.787 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 127, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.789 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 132, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 170, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 142, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.791 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 53, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.792 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 86, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.793 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 147, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 115, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 12, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.795 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 135, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.796 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 68, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.797 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 103, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.798 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 141, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 165, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 18, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.800 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 33, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.801 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 116, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.802 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 124, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 26, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 50, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.804 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 75, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 85, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 69, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 97, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 101, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.808 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 153, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.809 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 107, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.810 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 175, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.811 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 66, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.812 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 111, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 152, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 45, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.814 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 61, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.815 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 113, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.816 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 105, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 162, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 171, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.818 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 173, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 184, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.820 [Thread-15 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.afterAcquired:98 [aoy5vnfnjetz,6ek4wlg8c0ad]
> DEBUG - AfterAcquire 781 credits on address rpdm.v1.authentication,
> pendingCredits=63974
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.user.group.query
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.user.group.query, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.user.group.query
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.canvas.state.query
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.canvas.state.query, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.canvas.state.query
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:184 [z7wx0g93sdr7] DEBUG - not
> asking for -1 credits on acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.afterAcquired:98 [z7wx0g93sdr7] DEBUG -
> AfterAcquire 505 credits on address acf62840-864a-4eb4-a902-4861db94eb93,
> pendingCredits=65031
>
> 2024-01-10T19:52:34.824 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d43c72b6, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> ClientSessionFactoryImpl.java:410
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,dWlUyTFJQxRIqOG8poa5pxge5mYVJ48dUXfACv6LUtKsZzFSVGKyZagOScZPGCnB6cxsyzdobSyk8B6Bwd79504jcLncGDwqMeY-9K6dpOs,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> NettyConnector.java:835
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ZNE9b4nkKfo7M-_v-eXIZ86i5mBDuVpXhcuhH7d4Z3TXAYIWBA7iz2F7WJKgkGmWz9vMjcTKd3WNKgAqq_OrNuwfMP9rqwnZZdfDvOywCznI2b71&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:666
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,ud4sg1bGnGSsxR72jGNWKmRRRqC9RhLjc5Fa_mAH3joB4qQCFRRYETYRnfp_427BSgD3-id4mG56z7G-3qi9hl8xOglhqXamz4f_kpydTW4Drw,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,5ea91cHeAd9MIKFvy9sbzhVk4n_LaL3aA10YRDmCls04v12YLb2XMmXw6eBUcLVuIE_Z644AmEt_e0ljvQnrS_I6HavZOLh4zjfiHWUvesg,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,jDru8ObAZFnt6UJuRwWa5fbG9hi1-FGT2jjgjYcb0j9MnKP4NE8WTbKBHSHyKvKnv60VBWqMlES8saZuop8zduWr_U-bg4uKdFl24xth0Wc08-CvUBpd&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,FCoOt1eEzJLmEhVMnIIHAKa-kTFceMfQD6Kcwc4ZSxQI7u6ROHuAf16piW1PXfz5fwuSX1g87RD5aUJh-U4htAPPBkPWT1OqrJG8TvqS_LoMCTdPQPVU&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,RozNWdaPDnLkxzHk0PIIu7yOI0mq7J36SJtiYqHJSj83AaF6otFCOMT6S0SrKP0i5_HdKBbJFOfyhgDON-XzQcdSOvC2Z4N90Agiv9ZXimQSY87tDA,,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3xUOFzb6MCi6u1j2LRQL6CE5BWbu5eM6Ir7EsfF2oCaYUBtqnA1r3VDYEmgBiVBb7QypDxbOzY9knH1MLZZ3oikFpY_p69_553LHOTbV7g,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,9xoBaA3kGi2K8zQ8BuNLDq0dYmsnE1nuJw9sP_cDKKpUiUl1eHBvRBJF127g2yN9LNqLqZkyPVSq8YusAWP6k6MCI4_4n846oL1LY84Tow2q00oOukNL4Q,,&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,OqfGI8b5WUrx4fzs6Vhc7lLcb55MI-F2H4j9ic-9Fbs1_FhBfTLzdobmjbVxUHuNV9kcphYOQdlHoTf8bN7hu4yIeKsdhwLW0rasAMuEm4mWhNV8t2c,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,ml--e7XB5_TgtwWFh6lw23pXHYNiKpjFGKoUKmwrOwHyS3lnFQ78O6WUjq-OM1ITxXTrgD6P5K2E89TnqOgGcaOyn6aLaVIcvPhARMyQEqlFDI2mcnYwIfwADKB-&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,olHwwJjC2TUJmGxt3e1EpgRMfVMAAdmudKrqZ5ZByilJc51VAK_ZZHGL1mLqUqv59mtQqrIVOT0FHxcuVGnfmTsHrtP_Lfh6vEjgZmLr0Aa8EqAKSU0BDQ,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,3crdMkhGhlkrJXp-HsjZIEJOjIoajfvvI_g-uTOHgEc26-rm5spEwaDR4tTrIOKeN5HSQz3HfKcnanjUMfuquaMfvk7M7K2AtlUVxZdR9Q,,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,Qv7RvLuSjcsV2_72qHHLM6lmOI05RppEGGyR8280SVuKWQgOAl-EeFo8ZDeB_9Hw2f5ZgHbKDJbuB11rT-FBX6wf71ASJsIGNZYXzHn1HnNPy3pXaBaEgvrJvA,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,7hPEKeOrrW0p5qUGMytIu7avFNFhxNhXFzKgo6a-hDqm4ky01dphgxAxTpF5DmP3oUrVV1uWYwMbJuUFv1IoX_o9-H7Y6xXWdj067c1A&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,dU7lk32qJcXeW-cUtM1E65BB47JiLhSgJYTqhLIX373Pj0miueYi3kN7MYtAU5zT39qOtzzAiHcBVYWsGDB3N8R31seI_xbm-rQzfd0jzb6V1rSLy1kotiI,&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,BtR0iFYtXaxmAlygBDvC74BLAVELzJivsFTs_YS41zS_XagPPAr0UUKW1lNrafWE3WxaZcW_S5WbNn583_uHQ7aLAA9vskUJfxaMwUrM_DxAOd2N&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,nW_o_ayhGlEOytZThvhNsFjqKDq7oEaYAEjw_jVKEHAnXglYBlSGSA3yj7wyG-2DXRXiRZSD11UwsKx7fSGPuHkUUSoULIYqacVjSrfmUuEfG_A,&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:34.824 [Thread-19] JmsStaticConnectionPool.onException:78
> [] ERROR - Receive error occurred. javax.jms.JMSException:
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(
> ActiveMQConnection.java:714
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a714&c=E,1,3FhMyZ8H2FihvFzilmw5MqmH_tkqp5vHbCOJsN5x6sQNSLyFELuJt_Ej2M_24CH42Xx0iQesALGDJK8HHdR_PDaLploHa8R84-uwfftA3DUM&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(
> ActiveMQConnection.java:735
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a735&c=E,1,JvpZWdXDc673FDsIBWz8WgGu29y6j8H0zAczvf_UXYyUAzEMKtvXJ3tMkleaFsrHU-QdlAUi0bWddjnikuurWEEMfZs0jIDWZ3jnrx2__Vbg93yZ8oo,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(
> ClientSessionFactoryImpl.java:868
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a868&c=E,1,AznnL1cSXDTbCj6iQ4ovEeP71-oopbCcH0laxV9dB-Y_kBuerNfeOAXRkRLybAqgqQBjJZKBNFEoveCJjTGncMCguHnY0hvsdZuKIHz6BEhvRX7N&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(
> ClientSessionFactoryImpl.java:856
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a856&c=E,1,vBewrTh4rfp_ezwl572jqeiy0VBShHwQJNRT99yv4B54VM5Qaf2hTHrOq33emXwk_WSFrcueJSCKI90AxcN4jUznfP1ZLzL1dJdUz01zK4vN&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:802
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a802&c=E,1,EY-ry2DzdJloXWtaemW-dWnRy1-rr2mQFFjoCbmWjX6xHCNZF95ldY2zUgfhOhSQFfzm7eYV0PdRUTOwJrx9jCD4U-B5ztFqWmvqEmJQuqdCyQgypA,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,gHouFFMo7plejV1zi4ureD9QEDkW-WjvytRlH6DOyEt7oMO8ULkj6APov-gDhxtXAspTlk7qF6RYw-RrCqfSrXlcE0Cm45J61qJHt5Yc3zuBJJyEX8A,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,17MGSyeFWTwZKhb2DCqWeyFvdlinlKAunc4JSHJltjjiaCeT_1UjXFsTMtEiF_NYIOMI3brJbPkRW1QkJIhe_Br44jy1od8-BI3l37IfFJzZbcculrc,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,F6NyfiAiJI9fxHnOQnnpO7dLyoFavqwnNivvLL6x2LO54G8SpNhXjlUN748SkNWwe_oCD9JhI7u_BOk1uNbwSTYV0lXRyewBrwvvysSEOSCggiFCFRgjtA,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,TnFdrKl6nCZv9PR6lSsRZ2YN0r8LqpGQMRo3oHlgyE7_AgUMzrW12R4HeB5JhX8Akv-1iKeVGrnTLV2WeAbBOCM-fukhfRPsdUgq5pwe0yU,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,vAtPtyOuzaufOELKjMWnOBMrJEOShH0EzH9G3WL8gak4hWHmnVWPL1W7c_YnxRH2IerBQErg9VJApOiiX6TR6tcPau3xqRHrPyrsH70KHg2v&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,pak9EX0xrMkRTN71voUljZIOmUvksqdCCqWdwh8fq2R4OZXUQFIHNH3P9j0t-wpQStfSs_ZsT0a9BZYopaoyMSnMkN3xqvXn8A9dTehv&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,acJEflBFObRkFNBskLYk7AARfS1_CzZH3yFGYB9r_Ype7n0DAWduZhGpmImVc9k1pTnmFoA8L8UukMVvWBinxujU6r75ipmzh4I_cNQ5Gzmd97g5SAU4dRsf3Hw,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,aM5tVNMvOtLUxuoV8Dez1Y7fAS8KwME3peX3IqP_QPViIWgkZQahJW_1BMBQUo35u5h9hGJ3XSyDgkECwWOGDn6a9W-gctjh8WvUKx9yQ0sSLwheMcs5mhR2yVM,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,fUWcgLX3hupjGCjPYEByZ8xfzFx4iN-VnkLB9MarNu6OEOfrwnCkvJvtu6e-ZV-FREp0Ssisi_fezDu18eP9adS4mmcPPlJH69vnf0YDVg,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,KuAjbB9H9ddyi6unfXrVdNXylpB7l6qLIUfJARScyUnc7iH1GtvO0s1MAtB9fwkdqXq8idOJYZs_rg0iU-sN4aV24ID0Os5tsyvB34ba3ve2D3diROqixzkQuw4,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,EcYd8WT9GtMN9IXo58gsQpj5uNwcdCRtDfa0zG9t2jpvfdaSX5r9vLey2icJ-4wo3EyBMyUQzwyqYwC5BMfe4yHGjyo7h8jxmjcEKig4V0KyVa_SVq58NBT8&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,_a7OW7NorEoCm9eOnYjCi6TFl2__or_ickK-DwP4rrMmRX2Rj_TKaxpGjSAwCJZasllt1gefhShoob68MthMyp6bxv9N5fucG2HiF0g3yK4kqHTZjiFn7kxZ&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,-ttWHQzQDZ94E-HqKtYTKV2nuApkTk60Zy25iWFIykqfNe0o2seEul4UE_gETZr08B40bJjmZKMHq_-R0M8bQ8Fbu8twSkWx9MPLFUhc&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,624T4PuCyLmsS-UG0IoXRx3MMz4DqnU_1BuWfOz973WSpqfLsFFDAICu75KCoCwAs9O8WgKTEcBVqRhhf1JUqI3Znxem3LsG7Ml0Td3DnJJUoGYR0BE,&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7UL50c_ahB0IhSHbe9OlGABBKKE4OFyNM67SjmNrSo_LQhfViHAlLgvwdlRpbmDdymI5VaeOEklHjvlweLBz8Ske6wjc8J3CzM05VYenh6DzNiI,&typo=1&ancr_add=1>
> )
>
> Caused by: ActiveMQDisconnectedException[errorType=DISCONNECTED
> message=AMQ219015: The connection was disconnected because of server
> shutdown]
>
> ... 7 more
>
>
>
> 2024-01-10T19:52:34.828 [Thread-17 (ActiveMQ-client-global-threads)]
> ClientSessionImpl.acknowledge:1225 [] DEBUG - client ack messageID =
> 10054445
>
>
>
> 2024-01-10T19:52:34.862 [Thread-24 (ActiveMQ-client-global-threads)]
> JmsServerBase.logRequest:117 [] INFO - REQUEST IN:
>
>
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,01YVdf0IHIl4w2xIwUh5i7ToRiQfv6TxT-FRWSNW0n6e2aucHkWVyKxkYJTpG1jt7_EUCuQVS0NWGalDyf11AUIb3qVPj67IUtjJB484IdLKrHgbmSd-Q8Vg7AKZ&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Tuesday, January 9, 2024 1:57 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Both the "replicated-failback-static" and "replicated-failback" examples
> run without issue out of the box on my machine so I'm not sure what's going
> on with that. One thing that's odd is that out-of-the-box the line 122 [1]
> doesn't match the method call in your NPE. Perhaps you modified the example?
>
>
>
> Aside from that there were a handful of questions in my previous message
> you didn't address. I think the questions about thread dumps are the most
> pressing given that you indicated you're deleting the pod. My guess is that
> the client is getting hung in a network call that doesn't want to return
> due to the pod disappearing. A series of thread dumps during the minute
> delay would confirm or deny that.
>
>
>
>
>
> Justin
>
>
>
> [1]
> https://github.com/apache/activemq-artemis/blob/2.31.2/examples/features/ha/replicated-failback-static/src/main/java/org/apache/activemq/artemis/jms/example/ReplicatedFailbackStaticExample.java#L122
>
>
>
> On Tue, Jan 9, 2024 at 12:47 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hey Justing, thanks for the help.
>
>
>
> We are using version 2.31.2, which is pretty recent, if not the latest.
>
>
>
> With regards to the log traces, other AMQ threads show similar behavior,
> and our system isn’t doing much until AMQ-client stabilizes.
>
> I am running on a k3d cluster and the failover is triggered by me deleting
> the dm-activemq-live pod, which k8s then spins back up.
>
> Also, our slave config does NOT specify “allow-failback=true”, so the spin
> up of the new dm-activemq-live instance does not trigger a fail-back, which
> is intentional to eliminate failover-failback chaos in such a short time.
>
>
>
> I am trying to setup a simpler test case by using one of the examples,
> artemis/2.31.2/examples/features/ha/replicated-failback-static, however, I
> am getting this NPE on line 112:
>
> Caused by: java.lang.NullPointerException: Cannot invoke
> "javax.jms.TextMessage.getText()" because "message0" is null
>
>
>
> The NPE seems to indicate that, you cannot call consumer.receive() again
> even if NOT acknowledged, which seems to contradict the comments on the
> code. I have run into this same behavior in another example as well,
> artemis/2.31.2/examples/features/ha/replicated-failback.
>
>
>
> Can you please confirm the expected behavior of the example code in the
> 2.31.2 version? I can then modify the code accordingly. Or please point me
> to a working sample code.
>
>
>
> Thanks,
>
> Lino
>
>
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,lIOPl76F2R4EtenIAEJMeQ17kfmHQ6buC7G1W5TRefOcSfxvj_-3dLXV3MKDkCTkac_65Rw0SYmof66i91zGwgb_SS1nKaBpao_Lw7JNNk1mkE33&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Monday, January 8, 2024 9:20 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > Now, the backup process does start processing msgs right away...
>
>
>
> What evidence do you have of that?
>
>
>
> > ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute
> apart, before it stabilizes after the failover and is once again usable.
>
>
>
> The log snippets you provided show a reconnection attempt back to
> dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to
> dm-activemq-backup-svc:61617 both of which fail almost immediately. Then
> right after that another connection attempt is made to
> dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e.
> from 20:03:44.874 to 20:04:44.933). Are there any other logs during this
> minute? Did you happen to get any thread dumps during this time? If so,
> what did they show? If not, could you? What was happening with
> dm-activemq-live-svc during this time? Perhaps there was something weird
> going on with the network related to whatever you were doing with that
> machine to trigger failover.
>
>
>
> You might consider simplifying your test-case and then adding complexity
> bit by bit to see what triggers the difference in behavior. For example,
> run your test with the client and both brokers on the *same machine*. If
> that works then move just the client to a remote machine. If that works
> then test the client just reconnecting to the primary broker upon restart
> (i.e. no failover). Then finally put all three entities on separate
> machines and test failover. Hopefully such an exercise will give you a
> better understanding of where the problem might lie.
>
>
>
> Lastly, please indicate what version of ActiveMQ Artemis you're using.
>
>
>
>
>
> Justin
>
>
>
> On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hi Justin,
>
> I’m working with John Lilley on getting an ActiveMQ HA configuration
> operational. Our current HA config is replication with master and slave
> instances and with static connections, and it seems to be working, except
> that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect
> itself after a failover from the live to the backup instance. Now, the
> backup process does start processing msgs right away, but the
> ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart,
> before it stabilizes after the failover and is once again usable. I have
> included the relevant log traces from the Thread-0 ActiveMQ-client showing
> the Connection failure detection all the way through when the client is
> usable again, over 1 minute later. I am hoping that, based on the trace,
> you can find something that, perhaps, we should be doing differently in the
> client side?
>
>
>
> Here is our Client connection url:
>
> (tcp://dm-activemq-live-svc:
> 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
>
>
>
> Should we also be including the corresponding connector names?
>
>
>
> In an earlier email with John Lilley you recommended 2 brokers using
> “shared storage” instead of “replication”, which I will try next, however,
> since this issue seems to be on the client-side, I want to make sure our
> client-side is setup correctly.
>
>
>
> Thanks in advance for your help.
>
> Lino
>
>
>
> Thread-0 log traces:
>
> 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-live-svc/10.43.197.248:61616 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=a8a4a77c, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> ---
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and currentConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.197.248:61616
>
>
>
> 2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.11.76:61617
>
>
>
> 2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> ---
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and currentConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.197.248:61616
>
> 2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = ced14fbc
>
>
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.11.76:61617
>
>
>
> 2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
>
>
> 2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 10, performing as a failover operation now and recreating
> objects
>
>
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address
> activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,hqiYs6vnJzfq-X42G9W9ohJtYgn9LRPR-82RHxMyPnKUD5xBwCbo8W6b-PegfmAGwvEENlDO0k2IkoGFFegGxR-H6J72UHHlpEFbGtJzhCF3sucQ&typo=1&ancr_add=1>
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,KX9atZvl9XxpGPWobW9hXsmaLrVekfw2221zvD0PJ9Ik3fEQVyHKcZZCe7MQRciamxD-T-WDEk5Xa7_cAt0PSHwVW3jG7zzhq8XH9XIRC8EXmg,,&typo=1&ancr_add=1>,
> needed = 65536, arriving = 0
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,ZOCwWoQv34cYLuw9hZvJzBHfSrX6oQeQ_3-SetFMJa1hRC6lXgiKsAElOhfZ_vfnBhRGRJPZjSOqHIbwWO2ltSYYY7ElQjyItp0uDo79_jaDkw,,&typo=1&ancr_add=1>
>
> 2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=a8a4a77c, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> ClientSessionFactoryImpl.java:410
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,obbNTaFqn6mglgyZXy1S-0nBLNVtEYlkZBwAEKSGDGN0RuqlR_-7fIQPW4ELNqVFJdenAcKh65C3yQ2b_Lef7W01Tst6ZTrmI66KfHLae572jXv-HF3O&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> NettyConnector.java:835
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ytVVTXzfq-J1NXl2ct7T_47F5EbfZNQTXP8dlqw720EiCe32c6CLwlpk0-v4MzUD61FErq0nkRTalFpcTtIN8TDeBvz70rzb__TyeIDm48ZKfFU28qQ,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:666
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,KQYUuEgOaR6V8a_ZC_xBpKZmDneB4LTL1cQzn2Gqtla8ntHJ1fgRs9KelLgvbMd9zvBTg8maXUerYZYQZa-DlfK7fon0JZ0O74PNJ1SOCHqIPN4fZQ6Xv731OGOD&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,OSVh6NSORxz7qZ6eIuUDgWskoYfyXT4NNdjJYGMbH7HeWEUiymys2DnptjYKdmQfqjfHmI-egRZ31yWwd9MKzEd-gbbsuTUsqn_46niKaw,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,DM2boEBGg44zzcd0m37ITkOnfDrkqLE8ZSFGJpEI4AGo2GyOI2yKX08AEkKHxw_UnDMF1vccYEvESFDXjXBBjjIGO5TEsMLWZHbJF3AOIEQ73uc,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,wipgGeA7UVZIy1uuqvuxJEx4BA8TvYfSYmO617IgrvnOo-z7sn1vUjXFqIEU9M1HmedBVoHxB_LpvuEZ9p_biioPM8V-V9eS8i8oST46GvBhZZIc&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,0FNTjscT_d2CNIid0J8d_x7C7Peb6_neE-a_MZeKqkM1o_sPGujvpkXuiLv6kms9wSFyYqSux86C1MX2R8WKM8IZBsjKt7sCs03U2lRYlVbw2eTDg0ihSEV1nA,,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3yKPqxpIi9n43QJSCQiysKGlEYttZ5Dmuirc7jUAWNz6orLfIUCYwdDEJ3eU0QiHywsNAJdWqrvnx5rZK6-8TTJb6o8tr1WSvSL7omLF&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,6B4qLrj7q9DzDb0Trp9xgst7cOBhbD4hcEcGyPFul5E_mh3_oGmColB9YQkQTB1gJ6x5Mcm0VwvBLddpDpfFkcZwlzb4YNR9Ym5fcDclFW97pIrQ9oG4&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KqR93A_gEw1FpU7__PsYjSkOC5LH0zFTSKu5K9YlCdVAz3h_c9EkBndO7aliE4RSbK_qslEZp0b-YzJfCOxmB22jAfplsYxguMl1qjC0rWpNjfr9_fh4RjNL3AOK&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,mhChlKkihYjDw_Kk5SL1d_4eb7zhBlSzri_nwjOUsrs6MAvEfDAAjia07yTMUU6CQqc0WbDS2Fn8HJJI0UwJh1N7SKiz9dsP1gJLmgY66huLhXZE9c-3ECGXyIWQ&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,nn8IDmg6XQJLLXzH7kunXDNZa_3A7fUlnHdRCLoha7gQWJO1QeY32duGFAQMYaFHsfIbHbv142mei8HfCqRwrpuUfikYxLkFoXaD3pORZb9roBJs0H8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,DPLEvt-MPmmzv78LEkeXtYHM2q37m12TxX-nqjuIJurQdeXydsSakco9t_O7jgBbQ6y115xxdDb8IonFZbM86zgRDLcvUmeOSJuDmZybD_6JaTX0zpbQ&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,1vNFV-mDaLK92lJo0Mq6jvkCDzuwrSHT4lp1nB6KTzOCSO6comH5xNpEcvtZ_9pc0cQCD2kgDfB5xdxg0tAX2dF5oG5DHY6kxWXOBLnuAwiz1BkEFhw,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,aLydz0SGsclotqCro2upcTg44B_Ij7ckjApLu4B5JUwZKKdMo5z3RyUR4ikHFY8L3t_yB87Zp5atGRwoAPA1L9Mc9c681UjFrG16n-udVjeHtxQNxx8V-Nug&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1mZX-u-XHEZ_e4Ut7YETRvWm7U_qkC-q9QaPeHah4K4yNl5foyM1VSExCtYiisknkelDj6_K2MUHGc65Kd-6-JG_TXLby0OPWodJVjX3BtKaDNBUzusgpzT6ajL7&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-nbEPDxI7J-DFvbJfVVt2Cs1Vi_hN5bw-8vQF3SlM1nthw3VbJLWmS1h73cupUxqBoHCImTzDqI1m5gutx3ZriLpLpU7IqutHCqZuT6CTVWACPtj&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,-lUcdhDzaHWdAWLCc0IxWCFkKtKHxoEd64GWhzXAbeuG_Zm9mvn624BY6uS8MgQn-s2TMF8PbEPYVbOOy6BJuI8ByeiuAF9Y8f2I5wib&typo=1&ancr_add=1>
> )
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,T2UV-1dji1RODRcU4Ff4j_IzfTH2n47BatzyLZcEOU8VROFr3GS8re7CjDcnnHCSaPEtR-35J3z30dnYcz5K66rVsMVxgDiyh0FvEDKnN67naHjYbn0NpzAc&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
RE: Recommendations for HA/shared-store using Azure FileShare for the PV or performant alternatives
Posted by Vilius Šumskas <vi...@rivile.lt>.
Hi,
just thought to give my point of view on this.
Check if Azure NetApp NFS volume would provide better results for you. This is what we are using for our Artemis HA shared storage cluster on Google Cloud. It is more expensive but in our tests provided much better performance comparing to anything native Azure or GCP file stores give.
Also, I don’t want to encourage to drop K8s here, but to tell you the truth we abolished running Artemis on K8s all together. For our use case it was just too much work for too little gain. We run native Linux VMs with NetApp NFS attached storage. For the clients cluster is accessible by DNS directly without any additional load balancers. This way we left failover flow purely for the clients and the internal mechanics of the broker without the need to tackle K8s health checks, timeouts, graceful shutdowns and similar issues.
--
Best Regards,
Vilius Šumskas
Rivile
IT manager
From: Lino Pereira <li...@redpointglobal.com.INVALID>
Sent: Monday, January 15, 2024 10:10 PM
To: users@activemq.apache.org
Subject: Recommendations for HA/shared-store using Azure FileShare for the PV or performant alternatives
Hi Justin,
Upon testing our “shared-store” solution in our target Azure K8S cluster, we ran into performance issues, seemingly due to the use of an Azure FileShare for the shared-store PV. We are currently using "Azure FileShare NFS on Linux" and NIO for the journaling mode.
Azure FileShare info: https://learn.microsoft.com/en-us/azure/storage/files/files-nfs-protocol
What are your recommendations on choosing the best type of Azure FileShare to achieve the required performance?
Should we look at other journaling modes as a solution here?
Or, is going back to “replication”, with 2 backup instances to avoid split-brain, the best solution? Note that we do not need to persist messages for any longer than a couple of minutes.
Thanks,
Lino
[rg]<https://www.redpointglobal.com/>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Lino Pereira <li...@redpointglobal.com.INVALID>>
Sent: Thursday, January 11, 2024 4:11 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: RE: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hey Justin,
So I am trying to verify our AMQ docker images in the local environment, basically our system runs as simple processes on my local Linux system, but the AMQ instances run in docker containers. Now, this works fine for standalone, and for the HA case, the “master” instance works fine with, or without, the “backup” instance running. However, the “backup” instance doesn’t seem to ever start up fully, that is, it is missing the AMQ221080, AMQ221003, AMQ221007 and AMQ221001 log lines, shown below from the “master” instance. As a result, it never announces itself in “Backup” mode as seen when run from k8s. I am using the same broker configs that run fine when AMQ instances are run natively (no docker).
Also, when I stop the “master” instance, the AMQ-client reports these exceptions and never connects. Any ideas on why this may be happening for the backup, which uses the same exact docker image, just with a different broker.xml?
Is there some fundamental thing that I am missing when running 2 AMQ instances from docker containers?
My goal is to validate that our AMQ docker images are functional outside of k8s.
With regards, to getting you the thread dumps, next on the chopping block, can I just attach them to the email, or is there an alternate path to get those to you? Apologize for the dumb questions, but I am new at this.
Thanks.
Lino
AMQ-client exceptions after killing the “master” instance:
2024-01-11T15:47:15.322 [Thread-0 (ActiveMQ-client-global-threads)] client.errorCreatingNettyConnection:741 [] ERROR - AMQ214016: Failed to create netty connection java.lang.IllegalStateException: No ActiveMQChannelHandler has been found while connecting to localhost/127.0.0.1:61617 from Channel with id = 0bec947c
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:980<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a980&c=E,1,9sVwvAhg-dCUoWlCTkknJyr35Kb1lORNDVxR6-uqw3mazBhZ5pG19VN92yx8WmUo86gmEfiM-fyes3vKIRRqmUr7xGxTepKfWp8f8YCxIKDeU3MjhJIbTwA,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:866<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a866&c=E,1,PDO7x8ScbX2lkRczVyc7Qx5qFYh8jYI_QZSxIyD15vNJbt-HUSyFIAsL54Aas_i1kj247doxU1D-TF1f6UNtgifP4lOW0jVh-Ca8bmu8X2BRDUZyZYrlZd4,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:848<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a848&c=E,1,M5S848Ei6ypG34y6nHCsxBxodJIiZs0WdncK5b_Paenx_yRNXYZToN-BG2GJ4M8V03LsOGQzIKSbFW7uKF8az4WwG4Xrzk0hIvgvf05YYBR-KraWq2Quc6cO&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(ClientSessionFactoryImpl.java:1212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1212&c=E,1,Zu7gFKlm4HmSOURefrcnPu7_omfp3z6NkhUulfWOlvCON5qFl8o_m__R5jhaoNXCjQUWg3afJOkgibeky8zBPzqugdJrnx3mDcRvktMSa8mHPzzyhu9E&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1333<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1333&c=E,1,QLuXIqCAvTMp9vzA3xlqCzu6uu_CAV6-6B3ZjST9QtAqqlHPPW031VktMnXrspcOD4IBb1JKvR2F8OzNZIi97M8NdBmUvk48uDHiFLvxVM4,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1253<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1253&c=E,1,HGg4Q1mxe7PWvtXuKXcO1vTG9pGtPTivn_BSZ-cDFd_RB5T6KlxMhp2Z2iKrHRFFrcmAJyClu3xTgdygcYT1cRvfTIU0mnOfoNn73hij&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(ClientSessionFactoryImpl.java:1496<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1496&c=E,1,YDsqwgPbq8QPr_ODsKRUE5-FJdCxEgI2EdJdlE_ogGX_Dtkdqu0ZUW5FnNQHKMwQfJEjscTg0sXGDaY7D27_1M-BNdURAJybIJ0mM63IpSP_Kix0E7dJj0xAAA,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1074<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1074&c=E,1,UaHGLwmlvYxEp5Zvya3OgHoGpFPLrtGO8817O18DQ_eCtU9hppVQz3ZSqMOZR4Ou0RyfDnJ25Dsho8qeOiDziqkIZ-8smasmw6AS-SC3C07T9A,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:959<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a959&c=E,1,S2vAOnIQrU8itB1DHvtgyBp5ozbblk23_CGr3q-ohZ6Nt35WRISFB1PbqeDDdjRyaj8N9tzZSNUm_TpeEwNQg143NiFeTG18rXYlnq4jakQe2_H0HkGTrg5Qow,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:697<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a697&c=E,1,HRBbQntC7regy9ruiCBdgTxHOjjHiqqEtdrKtyuFoRluNvypXTYG7MHAazXY4rr-7iE9ymvG-ccC_EuBMTBG5AjelMg7KfPvqiIaUkF1RfMIvoeiA8o,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,WwOwlRDdAEBeA1Gg0ArmZFo1it0EZxIt6RsmaWdNGx6wo_Lz4qKdMxIHy7U_9S5bgl-33cfGzNtlYG9YM9FZXucb7-SjGMsXWWRiyGWMdHRJ&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:559<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a559&c=E,1,Zc7KWsYoeLt5E_P6hgERE60yuujJHXiWpuT9jFjj4gC367dihQ6w7V22qGaDAek1JHPJHKEsYT6vqwQAPm9GSln6px2RaJkIttMBpwOF398rKvWxlzAOmI4,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionException(ClientSessionFactoryImpl.java:425<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a425&c=E,1,BFL6YN5Kxy06vezMnsiL2nC8cdKvlemlxhvuHQ-FuWkoOzxoYzQuft1CeyeiQmEo-L40YbRsfc6dzmYWbWpQQb_xVyaLDbql7YwiFlA60a1xj1PSDmU,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$2.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f2.run&c=E,1,sMyuWkYLp3LX6ibrSEXRcAezwbBM7qAOluPp_GC1jVesMs6jxDzxY4FFM3VjeigvvheAyp3Q7Oe5ACoqft2B-abCTUa6WF1bt7Uh4xjE6hHhWxo,&typo=1&ancr_add=1>(NettyConnector.java:1252<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a1252&c=E,1,3JJSPZ_KDZLXcsGld2w64FmGt132PiKafjM_G1ZXb-ulOdS6Dwv6XUv8QHJqHN6CKkClQ7lUkNnakzibZdks8vAdD3gLbxCOyjzJGO6K745CLD2v2F7UK5zJSQGG&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,bpAKAylKI6Drp540buFl0pkNNpNnX4xnMjY3zWwpNyF_392duOqGQPjpdfCPGxrpBvc7_eIyfJpxl4WuufWnNYLFKdIg0Nu9WDi6STLxtiY0tUhT7eeO2iA,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,CqAtDgvRoDEnuFjXXv-RSPQ-NcQ5-m4IRpyQ9oiN0rhM40sqLFbhY9BMbc0HKKA9gNrRlaFd_DPaSB11LNOv4Rdahsd5fJLOqXHsZnvg3ShSJw,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,C5wuD-LRtJ7605p6qERurgxxNLnHBKbC4B7VE9lfDl7f_JsOdDSkeiyMkYACWnhSckLFmjUW2eFKZawLMNYORn4gTtULbgzlBoD8LWUyNto,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,rMwMgnKWBZ0Z_t5VCFvBOA7J5McS_4eMRlTXeB6c0yMri_CRxIy8PDWJRjrRtRMI1QpqaQtuBEcG4sCoeTP06BI4NmwaneyL2iZ_EcqSA0sDvg,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,AmU9mOKxf3Tw98cD6iOKGEUsELr-mALW0vP5N2eC_TxCQIG4d3rxa4xf_W2vXyThIPw3Zmp6MJGpD2c95V3s3JefMOVZJGioR7UrvoKjJHS1Qr22gQ,,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,uKLItaPZw48N90kGuOJ2giezmgZl49ttEaMZ2H4_Lr1mqtycK7JXSo7FWi90G25OhrIF9k-OxzV8LaDSYfMSQPd4uEksOAIsnz2-SSjTNFctPDccNQ,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,gIn498Y42-KQL16Kw7akU8ywkXBbJPMfxg7-Ivrxphmo6NJdJqsbcSdPEAdkwVYS3FgjewkEVfwxLXUtcBsU4QHUbsEaVo5060hl3qEBmZeRmUBspRNTPB0,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,KRj2MYUCdG64pKABEvud93N21Qgh8ET58T7SDN4ZzK9ErKl92F_UyzJtq4y9eDIqY04euQg_cRGOTddiPcrePTpRRyy9U7AEk_WKzWUcU_VDwdQacLRCEHF4j6ag&typo=1&ancr_add=1>)
-------------
Log excerpt from AMQ docker “master” instance, showing lines missing from backup:
2024-01-11 22:38:54,844 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST]
2024-01-11 22:38:54,853 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2024-01-11 22:38:54,905 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2024-01-11 22:38:54,906 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols [CORE,MQTT,AMQP,HORNETQ,STOMP,OPENWIRE]
2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.31.2 [live, nodeID=320ff011-b0d2-11ee-a54c-0242ac110004]
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,WIQnILFOGT12V7T2V5JgdPTF8Y-yZZKUo4gSIgdk9Xb0RKhobbQh-PjrSOCrn9J79hRQkhxWG4u7HGnntjzO4n-TgjsgxAzGPARQB9OM7Zk,&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Wednesday, January 10, 2024 6:57 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> ...I’ll have to instrument the code for that, right? Or am I missing an easier way to generate those on-demand?
I'm not clear on why you'd need to instrument the code to get thread dumps. You can generate thread dumps through a number of different commands, e.g.:
- jstack
- jcmd
- kill -3
You can execute these administratively or just set up a script that generates them every so often.
Justin
On Wed, Jan 10, 2024 at 7:43 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hey Justin,
I did finally get a chance to run everything locally, and it all works smoothly with failovers in both directions and the AMQ-client recovers within 2-3 seconds and with a lot less log noise.
I did not get the thread dumps yet because I’ll have to instrument the code for that, right? Or am I missing an easier way to generate those on-demand? This is happening in a k8s pod, so not easy to connect with jconsole.
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,P1-JTENl1pkpwfayJeVxTcbmyxoXqv5aWc8dgoVoQa2P9YJ6ITYanSwPPWI3OOvd-1ZuyvCfDdUgZSHNKSzXPNMT51ZJGff0MJI1JNYM4KnicxOXm0JmVYcKXz0s&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Wednesday, January 10, 2024 3:02 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> So our issue seems to be only for failover from live-to-backup, consistently.
As far as I know the client doesn't make any distinction between these two cases so I'm guessing there's some kind of environmental difference that's making an impact.
> Is there something that we are perhaps not doing on the client?
I don't think so. The connection URL you provided previously looks good.
> I will extend one of the example codes to use threads and sone of the AMQ-client wrapper code in an effort to isolate this behavior on a simpler code base, as you recommented.
Before going down that road it might be better to inspect the ostensibly low-hanging fruit first. Did you happen to test everything locally? If so, what happened? Did you acquire any thread dumps during the delay as previously requested?
Justin
On Wed, Jan 10, 2024 at 3:04 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hey Justin,
After starting fresh with version 2.31.2, the examples work out of the box for me as well.
And, after ensuring that version 2.31.2 is running everywhere, client was still using 2.30.0, and refactoring to using the recommended shared-store HA, our AMQ-client is still behaving as before. That is, failover from live-to-backup, client recovery takes over 1 minute, and logs report what I posted earlier. However, failover from backup-to-live, client recovers within 5 seconds, which is the expected behavior, and our system is happy. Please see new log excerpts with more logging included for the 2 relevant threads.
So our issue seems to be only for failover from live-to-backup, consistently. Is there something that we are perhaps not doing on the client?
I will extend one of the example codes to use threads and sone of the AMQ-client wrapper code in an effort to isolate this behavior on a simpler code base, as you recommented.
In the meantime, I thought I’d ping you in case you can think of something obvious that we might have missed, especially since we just moved from 2.30.0.
Thanks again,
Lino
Our versions:
ARTEMIS_VERSION=2.31.2
JMX_VERSION=0.18.0
Logs from failover from backup-to-live:
2024-01-10T19:52:29.588 [Thread-24 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d7ff7885, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,O0LJ4VfRCemWEaMLULajHsgUo6YseAVDyBlrTm2G50vM20WhlTUV0ueY1DxMRAMStXAoc6wy5Xx3x1vi2XsnE5F7_iNbL-MZ03M4IS9G&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,VdcejKqF9B67eWPDv-PP_0gR6PEmUxFYHHtW8kyn3Gzg3q-HHzL-Lr7oztjj2LQumPJWO-1KF4OPavU8c3DW4zPgrUsidhnnCpoMUUn_&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,xKCDuAN84vJBjsCp75epCz7V1lR05TFhETdvpPcKntBTPnQk0zuuQSei7-ly1RPUUsHDP8ZAe-IevqwGtTGfEB_yNzw90QcOpEx5BaXOW6E,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,pq7BenEb7b9i-FYz1LnTy-gHYDdV-28FiSv3pkV_i-1TQZtRxakbT-rOKkgLgz2IDx875Dk5EE_OrwsUcew9TIsvrcZZZ9ZIlwboxMEq9xnR5odpQL8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,8mmrZ5alRiVlyEshqjlGpedpt7TuUVBJOy8DDm2qHLGoY6v3pR76llV59sMsYwy2v4rwLqiD9Rcf3Hg4QSCZo7rdO1rQgMNapCKji2jg7T4IJOk3OAeLnIs,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,RLTYzCoyknTJ1GB9szMq3Jg02QAnt7juedv1HCTYmmA4JU1DO6MlJhkvzOTBqHbeeOdKSvwkualBrhGt0fUk7Nl5YLzlgfepqTWcDiBmXQ-_xSMcAHPw&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,iYtEGEWoGf58Vq3CGAxBrH3iWZ1jWm1N1b3AOFPNlx0kCahcxHEwXT7ZAQ_h3ynXTeKJSkvmBiGUkbqE3bH7tGw-Gn8VV7R7LaCkxl9Dch5I4kQDfJDKSg,,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,kH54RADAl8RcHDeCxJW9b-P4YJJFElj_Nbh6mHtmwlFdP2tKzzbK3wkAXg9tVqhIlsbd4gr2LJkn7DWZ4I-J8WVE-7UovVbVakuQTTxc2pbR9vuIDiEFztc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,V9-XUYgB7GoqPyGhEdE_CmtNood-yvXw6P9wRIqQOSYSWuTfiF1bhonQejBQfgJdHq8k2TsE1RAvcSOp_WzBE-ybl0l6rc5pIuMGdC1LUITpdk_Iuha7urSje4Y,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,RNgnFH7dBkD27TX353Mc-kqneOLcqFXEgpGywxg8VduTTYNyYFt5VqkKfirozQw9V88WOI8XK4v9dvrtPIl-3u7cp673GvKxq3L-XgqypuDtSnkRIYnh813U&typo=1&ancr_add=1>)
2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d43c72b6, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,L97BTn6AFLbUpYgHzAlQm9pDJ4Sn--TlGDpz05xOE3xd-qn9krfLJ6JzGTaiSpQwuVjuQ8KUPfUSW6N18pgw44-ay7eqYb2h0p3sxehZeXAC&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,PiedcDlDv-nxD-EF13k9j-sagD4clTOVvCs6xyDQvnrPzDUxc4xAbnHg0d7hnqeG4cVJZaO42LJvoNm59E3SmClgLAft-3N7F898HO6Obbkm&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,G5SZ7tGYpGBQzNRxdmIzsp4hWlUsmbY3_881k0N5lFoFBwuHt1dMCkKHI8wiZRGpS6qDY_UDBvDmKVQNRIHXIIyg5Df2nnWDCTLnbvWCO20fW0PC&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,MsagygznR7ie9qgJLlMogXq8ex-X4ST-fg_vbVbgcBanXur80DVykS1ftnzq0e4bWLE0k8EiNmlhlezemqTlBEncWMJz-AQAKdTxEk4WyVY,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,n7l9CUpZZbFH3ca7Iu2wxM3kNfged80mMRcu0Vn9wIoZz7aKzCP3c1OVR73iqF0oLbir7CjMHKAu3sfrR2cLs-EP7orqDWQKQApkzTZra9CJag,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,PePvk640nUsvwJWNJY39J2R954rSgiFVPPSQLOsjYskMxtDnDVBuzQiZ8ut5SAPoqzV9uYW6GX3ekOAAoou9eu6VpHa_oFc2-7d27W00VBLU_o_Zv4wEUw,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,REmfZe55ytZRsqZvFkCNsYUcRO5Yd8PZwIlQfGystV0qUeF2mXSYaWKgleI098Ac40PND4ilF-ukqEWDY8N04AIRQgGgKGpG036xx7jNSc78YS6UbKWiMNUHfNc,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,nnqfN76WKNs0ZYhVPzea17QO5aWkwmLqbfYrmQJ_qGQIgTzCL4PSXhjU9jy4mhbCUfl53cwF4ISYabr_zD1GUvCgv5m4dYW1_OROCBpe&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,vlJ5vYZAfrlVp9kL50a_jwWhqVW8-LMUgzeBb6mn-udcS_fMR1yld889bXgOXTMDiutFqKYX9XPh0RtxeXRLtnN31WqeOlH_CZv4_ZyhnOwXvm4J&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7Ba-VAkMm3x0E0f6F1f73TH_y9Ylo03hx9DEz4kqWzvrXy0gAv3D9zIM35eXIaB95UFxo72DkMpb0nKW_a-BIF89JobbRqjyVNLdC5K3L1EM520B&typo=1&ancr_add=1>)
2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:29.592 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:29.594 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:29.593 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:29.597 [Thread-6 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d4cf19d5
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.600 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 7b2392ac
2024-01-10T19:52:29.602 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.602 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.603 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = c1fb4c82
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.610 [Thread-8 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 06b7b3a3
2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:29.611 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 65680be4
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.624 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 97a109b7
2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.625 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.629 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 77b2de2f
2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:30.612 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = f92d7cd6
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.613 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 1d44895d
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.614 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = e8602acd
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:30.631 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 0a217bfd
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.632 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d2aa00d7
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.633 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 9a4f639e
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.634 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = ecdb1cbf
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:31.000 [Schedule-Runner] SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service as system
2024-01-10T19:52:32.001 [Schedule-Runner] SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service as system
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 2/200
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:32.116 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = a6b3d488
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.117 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = c0627cd9
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.118 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 71523eb3
2024-01-10T19:52:32.119 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.120 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:32.134 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 2/200
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:32.135 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 68efd0dc
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.137 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = daddef73
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.137 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 84813ce3
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.138 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 533990e9
2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:34.370 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 3/200
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:34.371 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 1d8df71b
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:34.373 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = de20e193
2024-01-10T19:52:34.373 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 3/200
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:34.390 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:34.390 [Thread-10 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 4661fd5d
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:34.392 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d1c14a74
2024-01-10T19:52:34.392 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.404 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-10T19:52:34.405 [Thread-9 (ActiveMQ-client-netty-threads)] ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up backup config = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc for live = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.405 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 38, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.409 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-10T19:52:34.410 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 37, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.411 [Thread-24 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 10, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.411 [Thread-11 (ActiveMQ-client-netty-threads)] ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up backup config = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc for live = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.413 [Thread-24 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,S5999ok8Om6JjNfBLhC0R2hYdl3dGqPWsLpis0MTx38AhKXPNqMMlH41RrDz0UZsxZzSZbLvUEOYFEa7hwM4N7gAZfCS9udcqSmOwhLg&typo=1&ancr_add=1>
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,CF5ZPcR0aBH6uaRjLjSeWoBKLnRwUOOgvrD5MzWuerKGIs1FEa5vZF8rdF1kWSKbLwGQlm1IsOeyey0f1t7eiA1ApUgs1EDhL7ilSV0c6rFxEanucXI,&typo=1&ancr_add=1>, needed = 65536, arriving = 0
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,nl9VlpvL5Lx7cxWUON3eMxxXV1my7fPq5WQLaWD1ozszhEPr1wVfqCwFilScj02MzRACloIhLZ8CgyMWoa_OqCu-Tf4folOdvKvtooRq&typo=1&ancr_add=1>
2024-01-10T19:52:34.415 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d7ff7885, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,awSP7U_0_MIm7Km9lwe59SrlpKsd9iUMA2y266CCeNZS6GJuEExdDYVS6EVMnlmPlVN0W1aqSx2Rss2Dm_YIM71s5j7OAUz6pAdGaTqnU_h2RyaoD6KyuoU,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,8gAKfG0MZn0OMdhCHJ98evOezuGoCS0mNbGWplSwawxhF0ed3tPq9vKWoWUbn2xmoLJ34skkCTpI2tUuevSCeVNT00tNpFlOJlyAjGWu&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,tXm2hjddgUkJac0QpMg4nVt2gwg5e3ORdql8K6xsJlBZIxph-NuPt1Ks2iB0ZarzoFELPAuAerVMiQpEU3nkq1JiqLAju26hE6Y2c3MwmKRTj66O8tsfQ-WFSh4d&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,P0xsXGcezpL-KzlLSOt_rfr6b6RF51B2hthJzXMGD04QeyLyg6AnMvF4M8ZOEwF90ojRdT3CtOy0x9HBSpGK_mmF_6Zl7Muz2TKrZRLAxQRxhV5XnvYlwQ,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,qctWvKkM1LAyegiROEl3hsiEN3dPcBcH4jlkBConOP-G2cgGl6X6XG-kdeQuBBOC8BmtTf-VmlbpJTjC_wznn8I6g7pLOg1uof2v0j_r3yyK7U8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,KRtNFryEe6jhqdT3of2FZC9PF7zGlYTxJ3KPFwO6TViZkyaHiybwHP3JmOhQ4F7yZtorUVRe5phyCswDWLc8lIFBVowIJygjwxLmO-F3xJSonSQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,ltH-LxKKSiKev4cs4CgS53B1Sn1z2t9IkxY_vjCKeoXMalJfF0yTujWPogG1a9qkWV5DrAT_qCTk7MdQa3LpkAae5yHTe_19fREF16p6vMMJqGXqLjw,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,66enqLb5-TVz_cB0ZCkDW4SpX4c3v0jslINMsxhHSjXfBWrIFoM8PxfSS7_VWmc_MdfmjPHIKcLm4zfcCe4MggwEWLxB-RjiUMqnmc8ShDJLoit5&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,xM_aKOMjg39ORE7gTvaGKP72ytDNvt6tjanfNTzekduYVGBvn963oijBiSY1JOQeorfgcCf57lVZSYIQ20x8sOFAH5zNexuTI0rBtrkDrtcJpq-S&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,Jr0OyBaxbJX-r7nzGvHtUkNxDkYucNHeuCPW_Mqw5I4IS0dve_XCjayhv7uvFQTdinHTsHqf2ZtZAn_qz1ZnE04PEsE0KWGfHQUj6N5WIKcNhyQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,fTod1kMxyNrLe6wmpPVQB1vco-7IrNIs9Dx8h9kWYDCH7xYH8V5kj2iao_X4D62DX5S24Xi08LA-xy2zHwj-LIzSKOA38iBBAlHdMD7qu3lK&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,37VGZcLM1KBT3tttPbLrhK822id0s0HCTimMYb18Jk-8bCbWxvPl2N9_yFFLgXa5pC9DXbKX6c8DMHolUFme8KRWVQdBx_lbRwtpLIOp6O7uiuV0fttkzvI3jxdO&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,zIgfFXrrGxOB_lKTtscVc_jXQlq_pH7TMnaK0IkixM4MVqoSgc-Wp5VV-QGQWmUtW5dggcgtYpRa7OoRHCBvDew7fxYJq28eSZVguCiuBiH9DzdKTHAXC5o,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,b6YutPqRZaL3jtcBxfqj_SiU4M_x_Qx-GeSmBfL7c6CG24IUL7AdjJxpyosXZtCtKco4bSDRxOj7HJn5CWgbgbPzJ0TFuXzJjs7VRUKVhg,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,xdw7r1BqGJuJPUUyo2a9wj_y2KKTC-j8c0DAocpHKzpZZaVOgPcxdng0Xi_JMKvlGYmcUvlTfZMm2FYQ7pTrF20xWUreNaRQqtU6rlWB3kkuPmMNBALyTBs,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,4iCAaiDF74c7cCKTdUIIlT6Qu6BmbQItK-gaAIrQtpNdie21ccXNS_dCydALepUtQNTgsVsVUs1NavHOugg0IlFK0CjTj8TKnThm55ffUeOx8zHc4oZUYpE_LRsz&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,qBGyQwtcrkAjVKJZ9cZzaW1ivhGQgGfhpcJUS_ww-vfcQVNfD9OjDwDX1XjpzJWYpo08MqSOrKLpgO8d72TBVtSGg2OQ4AmggCv3IUgKeA,,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,Mm6fpsoFsFLUtanrkJ6cqlk2y1zCJbPDIfqxsd0pS_sfS05oTAHBNIXIUUd4iA8HJY0guikOejZmykugHqKeopxLvJE3UmpX9YX1Y3Gu2ToK-B8aRw,,&typo=1&ancr_add=1>)
2024-01-10T19:52:34.435 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 54, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.465 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 98, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.474 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 20, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.483 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 128, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.495 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 158, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.499 [Thread-4] AbstractProducerCreditsImpl.checkCredits:184 [] DEBUG - not asking for -1 credits on activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,BYjREAStKM-0e-e5OOY86CxL1g9WrAP01m5ROYtsGCMHTh3wox28vU8_r7Y9e7QwRiDtJ065Xst_zM3u0oFx_K7Fr-ZkN1GBxPR6aWEKrm4nQA,,&typo=1&ancr_add=1>
2024-01-10T19:52:34.499 [Thread-4] AbstractProducerCreditsImpl.afterAcquired:98 [] DEBUG - AfterAcquire 333 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,r4o5lwz6oA1VbUpicHhTmEpC9U7BwWBMw8ZYMcrl4g-jVFykL22Cb2WJIZLOqv-hLbxRqgThYlmO40Jo84JsPHe-IYBjUU8WMQ-jy1lZqxq3YQk,&typo=1&ancr_add=1>, pendingCredits=65203
2024-01-10T19:52:34.506 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 106, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.680 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 155, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.683 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 21, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.684 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 129, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.686 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 57, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.687 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 174, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.688 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 176, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.689 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 56, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.690 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 108, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.691 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 156, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 233, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 112, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.693 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 187, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.694 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 110, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.695 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 144, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.696 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 235, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.697 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 28, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 47, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 30, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.699 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 77, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.701 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 65, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.710 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 70, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 34, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 159, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.713 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 76, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.714 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 119, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.716 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 59, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.719 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 87, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.720 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 80, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 166, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 149, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.722 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 137, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.723 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 163, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.724 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 179, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.726 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 181, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 17, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 58, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.728 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 32, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 49, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 234, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.730 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 143, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.731 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 42, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 136, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 178, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.733 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 55, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.734 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 95, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.737 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 48, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 16, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 146, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.739 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 22, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.742 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 43, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.743 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 130, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.744 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 36, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.745 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 60, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.746 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 117, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.751 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 14, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.752 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 81, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.754 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 93, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 138, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 100, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.756 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 62, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.757 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 125, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.758 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 15, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 99, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 44, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.760 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 172, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.761 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 84, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 79, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 122, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.764 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 64, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.765 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 88, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.767 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 151, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.772 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.773 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 96, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.774 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 27, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.775 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 73, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.776 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 90, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.777 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 134, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.778 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 35, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.780 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 118, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 186, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 41, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.782 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 31, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 150, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 145, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.787 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 127, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.789 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 132, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 170, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 142, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.791 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 53, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.792 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 86, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.793 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 147, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 115, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 12, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.795 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 135, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.796 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 68, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.797 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 103, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.798 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 141, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 165, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 18, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.800 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 33, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.801 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 116, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.802 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 124, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 26, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 50, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.804 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 75, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 85, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 69, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 97, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 101, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.808 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 153, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.809 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 107, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.810 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 175, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.811 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 66, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.812 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 111, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 152, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 45, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.814 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 61, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.815 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 113, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.816 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 105, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 162, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 171, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.818 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 173, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 184, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.820 [Thread-15 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.afterAcquired:98 [aoy5vnfnjetz,6ek4wlg8c0ad] DEBUG - AfterAcquire 781 credits on address rpdm.v1.authentication, pendingCredits=63974
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.user.group.query
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.user.group.query, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.user.group.query
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.canvas.state.query
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.canvas.state.query, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.canvas.state.query
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:184 [z7wx0g93sdr7] DEBUG - not asking for -1 credits on acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.afterAcquired:98 [z7wx0g93sdr7] DEBUG - AfterAcquire 505 credits on address acf62840-864a-4eb4-a902-4861db94eb93, pendingCredits=65031
2024-01-10T19:52:34.824 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d43c72b6, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,dWlUyTFJQxRIqOG8poa5pxge5mYVJ48dUXfACv6LUtKsZzFSVGKyZagOScZPGCnB6cxsyzdobSyk8B6Bwd79504jcLncGDwqMeY-9K6dpOs,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ZNE9b4nkKfo7M-_v-eXIZ86i5mBDuVpXhcuhH7d4Z3TXAYIWBA7iz2F7WJKgkGmWz9vMjcTKd3WNKgAqq_OrNuwfMP9rqwnZZdfDvOywCznI2b71&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,ud4sg1bGnGSsxR72jGNWKmRRRqC9RhLjc5Fa_mAH3joB4qQCFRRYETYRnfp_427BSgD3-id4mG56z7G-3qi9hl8xOglhqXamz4f_kpydTW4Drw,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,5ea91cHeAd9MIKFvy9sbzhVk4n_LaL3aA10YRDmCls04v12YLb2XMmXw6eBUcLVuIE_Z644AmEt_e0ljvQnrS_I6HavZOLh4zjfiHWUvesg,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,jDru8ObAZFnt6UJuRwWa5fbG9hi1-FGT2jjgjYcb0j9MnKP4NE8WTbKBHSHyKvKnv60VBWqMlES8saZuop8zduWr_U-bg4uKdFl24xth0Wc08-CvUBpd&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,FCoOt1eEzJLmEhVMnIIHAKa-kTFceMfQD6Kcwc4ZSxQI7u6ROHuAf16piW1PXfz5fwuSX1g87RD5aUJh-U4htAPPBkPWT1OqrJG8TvqS_LoMCTdPQPVU&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,RozNWdaPDnLkxzHk0PIIu7yOI0mq7J36SJtiYqHJSj83AaF6otFCOMT6S0SrKP0i5_HdKBbJFOfyhgDON-XzQcdSOvC2Z4N90Agiv9ZXimQSY87tDA,,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3xUOFzb6MCi6u1j2LRQL6CE5BWbu5eM6Ir7EsfF2oCaYUBtqnA1r3VDYEmgBiVBb7QypDxbOzY9knH1MLZZ3oikFpY_p69_553LHOTbV7g,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,9xoBaA3kGi2K8zQ8BuNLDq0dYmsnE1nuJw9sP_cDKKpUiUl1eHBvRBJF127g2yN9LNqLqZkyPVSq8YusAWP6k6MCI4_4n846oL1LY84Tow2q00oOukNL4Q,,&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,OqfGI8b5WUrx4fzs6Vhc7lLcb55MI-F2H4j9ic-9Fbs1_FhBfTLzdobmjbVxUHuNV9kcphYOQdlHoTf8bN7hu4yIeKsdhwLW0rasAMuEm4mWhNV8t2c,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,ml--e7XB5_TgtwWFh6lw23pXHYNiKpjFGKoUKmwrOwHyS3lnFQ78O6WUjq-OM1ITxXTrgD6P5K2E89TnqOgGcaOyn6aLaVIcvPhARMyQEqlFDI2mcnYwIfwADKB-&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,olHwwJjC2TUJmGxt3e1EpgRMfVMAAdmudKrqZ5ZByilJc51VAK_ZZHGL1mLqUqv59mtQqrIVOT0FHxcuVGnfmTsHrtP_Lfh6vEjgZmLr0Aa8EqAKSU0BDQ,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,3crdMkhGhlkrJXp-HsjZIEJOjIoajfvvI_g-uTOHgEc26-rm5spEwaDR4tTrIOKeN5HSQz3HfKcnanjUMfuquaMfvk7M7K2AtlUVxZdR9Q,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,Qv7RvLuSjcsV2_72qHHLM6lmOI05RppEGGyR8280SVuKWQgOAl-EeFo8ZDeB_9Hw2f5ZgHbKDJbuB11rT-FBX6wf71ASJsIGNZYXzHn1HnNPy3pXaBaEgvrJvA,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,7hPEKeOrrW0p5qUGMytIu7avFNFhxNhXFzKgo6a-hDqm4ky01dphgxAxTpF5DmP3oUrVV1uWYwMbJuUFv1IoX_o9-H7Y6xXWdj067c1A&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,dU7lk32qJcXeW-cUtM1E65BB47JiLhSgJYTqhLIX373Pj0miueYi3kN7MYtAU5zT39qOtzzAiHcBVYWsGDB3N8R31seI_xbm-rQzfd0jzb6V1rSLy1kotiI,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,BtR0iFYtXaxmAlygBDvC74BLAVELzJivsFTs_YS41zS_XagPPAr0UUKW1lNrafWE3WxaZcW_S5WbNn583_uHQ7aLAA9vskUJfxaMwUrM_DxAOd2N&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,nW_o_ayhGlEOytZThvhNsFjqKDq7oEaYAEjw_jVKEHAnXglYBlSGSA3yj7wyG-2DXRXiRZSD11UwsKx7fSGPuHkUUSoULIYqacVjSrfmUuEfG_A,&typo=1&ancr_add=1>)
2024-01-10T19:52:34.824 [Thread-19] JmsStaticConnectionPool.onException:78 [] ERROR - Receive error occurred. javax.jms.JMSException: ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:714<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a714&c=E,1,3FhMyZ8H2FihvFzilmw5MqmH_tkqp5vHbCOJsN5x6sQNSLyFELuJt_Ej2M_24CH42Xx0iQesALGDJK8HHdR_PDaLploHa8R84-uwfftA3DUM&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:735<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a735&c=E,1,JvpZWdXDc673FDsIBWz8WgGu29y6j8H0zAczvf_UXYyUAzEMKtvXJ3tMkleaFsrHU-QdlAUi0bWddjnikuurWEEMfZs0jIDWZ3jnrx2__Vbg93yZ8oo,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:868<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a868&c=E,1,AznnL1cSXDTbCj6iQ4ovEeP71-oopbCcH0laxV9dB-Y_kBuerNfeOAXRkRLybAqgqQBjJZKBNFEoveCJjTGncMCguHnY0hvsdZuKIHz6BEhvRX7N&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:856<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a856&c=E,1,vBewrTh4rfp_ezwl572jqeiy0VBShHwQJNRT99yv4B54VM5Qaf2hTHrOq33emXwk_WSFrcueJSCKI90AxcN4jUznfP1ZLzL1dJdUz01zK4vN&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:802<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a802&c=E,1,EY-ry2DzdJloXWtaemW-dWnRy1-rr2mQFFjoCbmWjX6xHCNZF95ldY2zUgfhOhSQFfzm7eYV0PdRUTOwJrx9jCD4U-B5ztFqWmvqEmJQuqdCyQgypA,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,gHouFFMo7plejV1zi4ureD9QEDkW-WjvytRlH6DOyEt7oMO8ULkj6APov-gDhxtXAspTlk7qF6RYw-RrCqfSrXlcE0Cm45J61qJHt5Yc3zuBJJyEX8A,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,17MGSyeFWTwZKhb2DCqWeyFvdlinlKAunc4JSHJltjjiaCeT_1UjXFsTMtEiF_NYIOMI3brJbPkRW1QkJIhe_Br44jy1od8-BI3l37IfFJzZbcculrc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,F6NyfiAiJI9fxHnOQnnpO7dLyoFavqwnNivvLL6x2LO54G8SpNhXjlUN748SkNWwe_oCD9JhI7u_BOk1uNbwSTYV0lXRyewBrwvvysSEOSCggiFCFRgjtA,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,TnFdrKl6nCZv9PR6lSsRZ2YN0r8LqpGQMRo3oHlgyE7_AgUMzrW12R4HeB5JhX8Akv-1iKeVGrnTLV2WeAbBOCM-fukhfRPsdUgq5pwe0yU,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,vAtPtyOuzaufOELKjMWnOBMrJEOShH0EzH9G3WL8gak4hWHmnVWPL1W7c_YnxRH2IerBQErg9VJApOiiX6TR6tcPau3xqRHrPyrsH70KHg2v&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,pak9EX0xrMkRTN71voUljZIOmUvksqdCCqWdwh8fq2R4OZXUQFIHNH3P9j0t-wpQStfSs_ZsT0a9BZYopaoyMSnMkN3xqvXn8A9dTehv&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,acJEflBFObRkFNBskLYk7AARfS1_CzZH3yFGYB9r_Ype7n0DAWduZhGpmImVc9k1pTnmFoA8L8UukMVvWBinxujU6r75ipmzh4I_cNQ5Gzmd97g5SAU4dRsf3Hw,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,aM5tVNMvOtLUxuoV8Dez1Y7fAS8KwME3peX3IqP_QPViIWgkZQahJW_1BMBQUo35u5h9hGJ3XSyDgkECwWOGDn6a9W-gctjh8WvUKx9yQ0sSLwheMcs5mhR2yVM,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,fUWcgLX3hupjGCjPYEByZ8xfzFx4iN-VnkLB9MarNu6OEOfrwnCkvJvtu6e-ZV-FREp0Ssisi_fezDu18eP9adS4mmcPPlJH69vnf0YDVg,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,KuAjbB9H9ddyi6unfXrVdNXylpB7l6qLIUfJARScyUnc7iH1GtvO0s1MAtB9fwkdqXq8idOJYZs_rg0iU-sN4aV24ID0Os5tsyvB34ba3ve2D3diROqixzkQuw4,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,EcYd8WT9GtMN9IXo58gsQpj5uNwcdCRtDfa0zG9t2jpvfdaSX5r9vLey2icJ-4wo3EyBMyUQzwyqYwC5BMfe4yHGjyo7h8jxmjcEKig4V0KyVa_SVq58NBT8&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,_a7OW7NorEoCm9eOnYjCi6TFl2__or_ickK-DwP4rrMmRX2Rj_TKaxpGjSAwCJZasllt1gefhShoob68MthMyp6bxv9N5fucG2HiF0g3yK4kqHTZjiFn7kxZ&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,-ttWHQzQDZ94E-HqKtYTKV2nuApkTk60Zy25iWFIykqfNe0o2seEul4UE_gETZr08B40bJjmZKMHq_-R0M8bQ8Fbu8twSkWx9MPLFUhc&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,624T4PuCyLmsS-UG0IoXRx3MMz4DqnU_1BuWfOz973WSpqfLsFFDAICu75KCoCwAs9O8WgKTEcBVqRhhf1JUqI3Znxem3LsG7Ml0Td3DnJJUoGYR0BE,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7UL50c_ahB0IhSHbe9OlGABBKKE4OFyNM67SjmNrSo_LQhfViHAlLgvwdlRpbmDdymI5VaeOEklHjvlweLBz8Ske6wjc8J3CzM05VYenh6DzNiI,&typo=1&ancr_add=1>)
Caused by: ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
... 7 more
2024-01-10T19:52:34.828 [Thread-17 (ActiveMQ-client-global-threads)] ClientSessionImpl.acknowledge:1225 [] DEBUG - client ack messageID = 10054445
2024-01-10T19:52:34.862 [Thread-24 (ActiveMQ-client-global-threads)] JmsServerBase.logRequest:117 [] INFO - REQUEST IN:
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,01YVdf0IHIl4w2xIwUh5i7ToRiQfv6TxT-FRWSNW0n6e2aucHkWVyKxkYJTpG1jt7_EUCuQVS0NWGalDyf11AUIb3qVPj67IUtjJB484IdLKrHgbmSd-Q8Vg7AKZ&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Tuesday, January 9, 2024 1:57 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Both the "replicated-failback-static" and "replicated-failback" examples run without issue out of the box on my machine so I'm not sure what's going on with that. One thing that's odd is that out-of-the-box the line 122 [1] doesn't match the method call in your NPE. Perhaps you modified the example?
Aside from that there were a handful of questions in my previous message you didn't address. I think the questions about thread dumps are the most pressing given that you indicated you're deleting the pod. My guess is that the client is getting hung in a network call that doesn't want to return due to the pod disappearing. A series of thread dumps during the minute delay would confirm or deny that.
Justin
[1] https://github.com/apache/activemq-artemis/blob/2.31.2/examples/features/ha/replicated-failback-static/src/main/java/org/apache/activemq/artemis/jms/example/ReplicatedFailbackStaticExample.java#L122
On Tue, Jan 9, 2024 at 12:47 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hey Justing, thanks for the help.
We are using version 2.31.2, which is pretty recent, if not the latest.
With regards to the log traces, other AMQ threads show similar behavior, and our system isn’t doing much until AMQ-client stabilizes.
I am running on a k3d cluster and the failover is triggered by me deleting the dm-activemq-live pod, which k8s then spins back up.
Also, our slave config does NOT specify “allow-failback=true”, so the spin up of the new dm-activemq-live instance does not trigger a fail-back, which is intentional to eliminate failover-failback chaos in such a short time.
I am trying to setup a simpler test case by using one of the examples, artemis/2.31.2/examples/features/ha/replicated-failback-static, however, I am getting this NPE on line 112:
Caused by: java.lang.NullPointerException: Cannot invoke "javax.jms.TextMessage.getText()" because "message0" is null
The NPE seems to indicate that, you cannot call consumer.receive() again even if NOT acknowledged, which seems to contradict the comments on the code. I have run into this same behavior in another example as well, artemis/2.31.2/examples/features/ha/replicated-failback.
Can you please confirm the expected behavior of the example code in the 2.31.2 version? I can then modify the code accordingly. Or please point me to a working sample code.
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,lIOPl76F2R4EtenIAEJMeQ17kfmHQ6buC7G1W5TRefOcSfxvj_-3dLXV3MKDkCTkac_65Rw0SYmof66i91zGwgb_SS1nKaBpao_Lw7JNNk1mkE33&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Monday, January 8, 2024 9:20 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> Now, the backup process does start processing msgs right away...
What evidence do you have of that?
> ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable.
The log snippets you provided show a reconnection attempt back to dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to dm-activemq-backup-svc:61617 both of which fail almost immediately. Then right after that another connection attempt is made to dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e. from 20:03:44.874 to 20:04:44.933). Are there any other logs during this minute? Did you happen to get any thread dumps during this time? If so, what did they show? If not, could you? What was happening with dm-activemq-live-svc during this time? Perhaps there was something weird going on with the network related to whatever you were doing with that machine to trigger failover.
You might consider simplifying your test-case and then adding complexity bit by bit to see what triggers the difference in behavior. For example, run your test with the client and both brokers on the *same machine*. If that works then move just the client to a remote machine. If that works then test the client just reconnecting to the primary broker upon restart (i.e. no failover). Then finally put all three entities on separate machines and test failover. Hopefully such an exercise will give you a better understanding of where the problem might lie.
Lastly, please indicate what version of ActiveMQ Artemis you're using.
Justin
On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
I’m working with John Lilley on getting an ActiveMQ HA configuration operational. Our current HA config is replication with master and slave instances and with static connections, and it seems to be working, except that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect itself after a failover from the live to the backup instance. Now, the backup process does start processing msgs right away, but the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable. I have included the relevant log traces from the Thread-0 ActiveMQ-client showing the Connection failure detection all the way through when the client is usable again, over 1 minute later. I am hoping that, based on the trace, you can find something that, perhaps, we should be doing differently in the client side?
Here is our Client connection url:
(tcp://dm-activemq-live-svc: 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
Should we also be including the corresponding connector names?
In an earlier email with John Lilley you recommended 2 brokers using “shared storage” instead of “replication”, which I will try next, however, since this issue seems to be on the client-side, I want to make sure our client-side is setup correctly.
Thanks in advance for your help.
Lino
Thread-0 log traces:
2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
---
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616>
2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617<http://10.43.11.76:61617>
2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
---
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616>
2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = ced14fbc
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617<http://10.43.11.76:61617>
2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 10, performing as a failover operation now and recreating objects
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,hqiYs6vnJzfq-X42G9W9ohJtYgn9LRPR-82RHxMyPnKUD5xBwCbo8W6b-PegfmAGwvEENlDO0k2IkoGFFegGxR-H6J72UHHlpEFbGtJzhCF3sucQ&typo=1&ancr_add=1>
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,KX9atZvl9XxpGPWobW9hXsmaLrVekfw2221zvD0PJ9Ik3fEQVyHKcZZCe7MQRciamxD-T-WDEk5Xa7_cAt0PSHwVW3jG7zzhq8XH9XIRC8EXmg,,&typo=1&ancr_add=1>, needed = 65536, arriving = 0
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,ZOCwWoQv34cYLuw9hZvJzBHfSrX6oQeQ_3-SetFMJa1hRC6lXgiKsAElOhfZ_vfnBhRGRJPZjSOqHIbwWO2ltSYYY7ElQjyItp0uDo79_jaDkw,,&typo=1&ancr_add=1>
2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,obbNTaFqn6mglgyZXy1S-0nBLNVtEYlkZBwAEKSGDGN0RuqlR_-7fIQPW4ELNqVFJdenAcKh65C3yQ2b_Lef7W01Tst6ZTrmI66KfHLae572jXv-HF3O&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ytVVTXzfq-J1NXl2ct7T_47F5EbfZNQTXP8dlqw720EiCe32c6CLwlpk0-v4MzUD61FErq0nkRTalFpcTtIN8TDeBvz70rzb__TyeIDm48ZKfFU28qQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,KQYUuEgOaR6V8a_ZC_xBpKZmDneB4LTL1cQzn2Gqtla8ntHJ1fgRs9KelLgvbMd9zvBTg8maXUerYZYQZa-DlfK7fon0JZ0O74PNJ1SOCHqIPN4fZQ6Xv731OGOD&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,OSVh6NSORxz7qZ6eIuUDgWskoYfyXT4NNdjJYGMbH7HeWEUiymys2DnptjYKdmQfqjfHmI-egRZ31yWwd9MKzEd-gbbsuTUsqn_46niKaw,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,DM2boEBGg44zzcd0m37ITkOnfDrkqLE8ZSFGJpEI4AGo2GyOI2yKX08AEkKHxw_UnDMF1vccYEvESFDXjXBBjjIGO5TEsMLWZHbJF3AOIEQ73uc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,wipgGeA7UVZIy1uuqvuxJEx4BA8TvYfSYmO617IgrvnOo-z7sn1vUjXFqIEU9M1HmedBVoHxB_LpvuEZ9p_biioPM8V-V9eS8i8oST46GvBhZZIc&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,0FNTjscT_d2CNIid0J8d_x7C7Peb6_neE-a_MZeKqkM1o_sPGujvpkXuiLv6kms9wSFyYqSux86C1MX2R8WKM8IZBsjKt7sCs03U2lRYlVbw2eTDg0ihSEV1nA,,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3yKPqxpIi9n43QJSCQiysKGlEYttZ5Dmuirc7jUAWNz6orLfIUCYwdDEJ3eU0QiHywsNAJdWqrvnx5rZK6-8TTJb6o8tr1WSvSL7omLF&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,6B4qLrj7q9DzDb0Trp9xgst7cOBhbD4hcEcGyPFul5E_mh3_oGmColB9YQkQTB1gJ6x5Mcm0VwvBLddpDpfFkcZwlzb4YNR9Ym5fcDclFW97pIrQ9oG4&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KqR93A_gEw1FpU7__PsYjSkOC5LH0zFTSKu5K9YlCdVAz3h_c9EkBndO7aliE4RSbK_qslEZp0b-YzJfCOxmB22jAfplsYxguMl1qjC0rWpNjfr9_fh4RjNL3AOK&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,mhChlKkihYjDw_Kk5SL1d_4eb7zhBlSzri_nwjOUsrs6MAvEfDAAjia07yTMUU6CQqc0WbDS2Fn8HJJI0UwJh1N7SKiz9dsP1gJLmgY66huLhXZE9c-3ECGXyIWQ&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,nn8IDmg6XQJLLXzH7kunXDNZa_3A7fUlnHdRCLoha7gQWJO1QeY32duGFAQMYaFHsfIbHbv142mei8HfCqRwrpuUfikYxLkFoXaD3pORZb9roBJs0H8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,DPLEvt-MPmmzv78LEkeXtYHM2q37m12TxX-nqjuIJurQdeXydsSakco9t_O7jgBbQ6y115xxdDb8IonFZbM86zgRDLcvUmeOSJuDmZybD_6JaTX0zpbQ&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,1vNFV-mDaLK92lJo0Mq6jvkCDzuwrSHT4lp1nB6KTzOCSO6comH5xNpEcvtZ_9pc0cQCD2kgDfB5xdxg0tAX2dF5oG5DHY6kxWXOBLnuAwiz1BkEFhw,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,aLydz0SGsclotqCro2upcTg44B_Ij7ckjApLu4B5JUwZKKdMo5z3RyUR4ikHFY8L3t_yB87Zp5atGRwoAPA1L9Mc9c681UjFrG16n-udVjeHtxQNxx8V-Nug&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1mZX-u-XHEZ_e4Ut7YETRvWm7U_qkC-q9QaPeHah4K4yNl5foyM1VSExCtYiisknkelDj6_K2MUHGc65Kd-6-JG_TXLby0OPWodJVjX3BtKaDNBUzusgpzT6ajL7&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-nbEPDxI7J-DFvbJfVVt2Cs1Vi_hN5bw-8vQF3SlM1nthw3VbJLWmS1h73cupUxqBoHCImTzDqI1m5gutx3ZriLpLpU7IqutHCqZuT6CTVWACPtj&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,-lUcdhDzaHWdAWLCc0IxWCFkKtKHxoEd64GWhzXAbeuG_Zm9mvn624BY6uS8MgQn-s2TMF8PbEPYVbOOy6BJuI8ByeiuAF9Y8f2I5wib&typo=1&ancr_add=1>)
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,T2UV-1dji1RODRcU4Ff4j_IzfTH2n47BatzyLZcEOU8VROFr3GS8re7CjDcnnHCSaPEtR-35J3z30dnYcz5K66rVsMVxgDiyh0FvEDKnN67naHjYbn0NpzAc&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Recommendations for HA/shared-store using Azure FileShare for the PV or performant alternatives
Posted by Justin Bertram <jb...@apache.org>.
> What are your recommendations on choosing the best type of Azure
FileShare to achieve the required performance?
It's not clear what the current performance is or what your performance
goal is. If you want to stick with shared storage then I can only recommend
you work with Azure to increase performance since this is in their
environment and presumably they control hardware and service levels.
> Should we look at other journaling modes as a solution here?
If you can't ultimately meet your performance goals with shared storage
then you should definitely look at other options.
> Or, is going back to “replication”, with 2 backup instances to avoid
split-brain, the best solution? Note that we do not need to persist
messages for any longer than a couple of minutes.
Two backup instances with replication will not mitigate split-brain. You'd
need at least 3 HA pairs of brokers to establish a proper quorum or you
could use a single HA pair and integrate with, e.g. ZooKeeper.
For what it's worth, I recently sent a PR [1] to clarify much of the
documentation around HA - specifically in regard to split-brain.
Justin
[1] https://github.com/apache/activemq-artemis/pull/4735
On Mon, Jan 15, 2024 at 2:11 PM Lino Pereira
<li...@redpointglobal.com.invalid> wrote:
> Hi Justin,
>
>
>
> Upon testing our “shared-store” solution in our target Azure K8S cluster,
> we ran into performance issues, seemingly due to the use of an Azure
> FileShare for the shared-store PV. We are currently using "Azure
> FileShare NFS on Linux" and NIO for the journaling mode.
>
>
>
> Azure FileShare info:
> https://learn.microsoft.com/en-us/azure/storage/files/files-nfs-protocol
>
>
>
> What are your recommendations on choosing the best type of Azure FileShare
> to achieve the required performance?
>
>
>
> Should we look at other journaling modes as a solution here?
>
>
>
> Or, is going back to “replication”, with 2 backup instances to avoid
> split-brain, the best solution? Note that we do not need to persist
> messages for any longer than a couple of minutes.
>
>
>
> Thanks,
>
> Lino
>
>
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> Lino Pereira
>
> C++ Developer, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Lino Pereira <li...@redpointglobal.com.INVALID>
> *Sent:* Thursday, January 11, 2024 4:11 PM
> *To:* users@activemq.apache.org
> *Subject:* RE: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Hey Justin,
>
>
>
> So I am trying to verify our AMQ docker images in the local environment,
> basically our system runs as simple processes on my local Linux system, but
> the AMQ instances run in docker containers. Now, this works fine for
> standalone, and for the HA case, the “master” instance works fine with, or
> without, the “backup” instance running. However, the “backup” instance
> doesn’t seem to ever start up fully, that is, it is missing the AMQ221080,
> AMQ221003, AMQ221007 and AMQ221001 log lines, shown below from the
> “master” instance. As a result, it never announces itself in “Backup” mode
> as seen when run from k8s. I am using the same broker configs that run fine
> when AMQ instances are run natively (no docker).
>
>
>
> Also, when I stop the “master” instance, the AMQ-client reports these
> exceptions and never connects. Any ideas on why this may be happening for
> the backup, which uses the same exact docker image, just with a different
> broker.xml?
>
> Is there some fundamental thing that I am missing when running 2 AMQ
> instances from docker containers?
>
> My goal is to validate that our AMQ docker images are functional outside
> of k8s.
>
>
>
> With regards, to getting you the thread dumps, next on the chopping block,
> can I just attach them to the email, or is there an alternate path to get
> those to you? Apologize for the dumb questions, but I am new at this.
>
>
>
> Thanks.
>
> Lino
>
>
>
> AMQ-client exceptions after killing the “master” instance:
>
> 2024-01-11T15:47:15.322 [Thread-0 (ActiveMQ-client-global-threads)]
> client.errorCreatingNettyConnection:741 [] ERROR - AMQ214016: Failed to
> create netty connection java.lang.IllegalStateException: No
> ActiveMQChannelHandler has been found while connecting to localhost/
> 127.0.0.1:61617 from Channel with id = 0bec947c
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(
> NettyConnector.java:980
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a980&c=E,1,9sVwvAhg-dCUoWlCTkknJyr35Kb1lORNDVxR6-uqw3mazBhZ5pG19VN92yx8WmUo86gmEfiM-fyes3vKIRRqmUr7xGxTepKfWp8f8YCxIKDeU3MjhJIbTwA,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(
> NettyConnector.java:866
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a866&c=E,1,PDO7x8ScbX2lkRczVyc7Qx5qFYh8jYI_QZSxIyD15vNJbt-HUSyFIAsL54Aas_i1kj247doxU1D-TF1f6UNtgifP4lOW0jVh-Ca8bmu8X2BRDUZyZYrlZd4,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(
> NettyConnector.java:848
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a848&c=E,1,M5S848Ei6ypG34y6nHCsxBxodJIiZs0WdncK5b_Paenx_yRNXYZToN-BG2GJ4M8V03LsOGQzIKSbFW7uKF8az4WwG4Xrzk0hIvgvf05YYBR-KraWq2Quc6cO&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(
> ClientSessionFactoryImpl.java:1212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1212&c=E,1,Zu7gFKlm4HmSOURefrcnPu7_omfp3z6NkhUulfWOlvCON5qFl8o_m__R5jhaoNXCjQUWg3afJOkgibeky8zBPzqugdJrnx3mDcRvktMSa8mHPzzyhu9E&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(
> ClientSessionFactoryImpl.java:1333
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1333&c=E,1,QLuXIqCAvTMp9vzA3xlqCzu6uu_CAV6-6B3ZjST9QtAqqlHPPW031VktMnXrspcOD4IBb1JKvR2F8OzNZIi97M8NdBmUvk48uDHiFLvxVM4,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(
> ClientSessionFactoryImpl.java:1253
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1253&c=E,1,HGg4Q1mxe7PWvtXuKXcO1vTG9pGtPTivn_BSZ-cDFd_RB5T6KlxMhp2Z2iKrHRFFrcmAJyClu3xTgdygcYT1cRvfTIU0mnOfoNn73hij&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(
> ClientSessionFactoryImpl.java:1496
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1496&c=E,1,YDsqwgPbq8QPr_ODsKRUE5-FJdCxEgI2EdJdlE_ogGX_Dtkdqu0ZUW5FnNQHKMwQfJEjscTg0sXGDaY7D27_1M-BNdURAJybIJ0mM63IpSP_Kix0E7dJj0xAAA,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(
> ClientSessionFactoryImpl.java:1074
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1074&c=E,1,UaHGLwmlvYxEp5Zvya3OgHoGpFPLrtGO8817O18DQ_eCtU9hppVQz3ZSqMOZR4Ou0RyfDnJ25Dsho8qeOiDziqkIZ-8smasmw6AS-SC3C07T9A,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(
> ClientSessionFactoryImpl.java:959
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a959&c=E,1,S2vAOnIQrU8itB1DHvtgyBp5ozbblk23_CGr3q-ohZ6Nt35WRISFB1PbqeDDdjRyaj8N9tzZSNUm_TpeEwNQg143NiFeTG18rXYlnq4jakQe2_H0HkGTrg5Qow,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:697
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a697&c=E,1,HRBbQntC7regy9ruiCBdgTxHOjjHiqqEtdrKtyuFoRluNvypXTYG7MHAazXY4rr-7iE9ymvG-ccC_EuBMTBG5AjelMg7KfPvqiIaUkF1RfMIvoeiA8o,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,WwOwlRDdAEBeA1Gg0ArmZFo1it0EZxIt6RsmaWdNGx6wo_Lz4qKdMxIHy7U_9S5bgl-33cfGzNtlYG9YM9FZXucb7-SjGMsXWWRiyGWMdHRJ&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:559
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a559&c=E,1,Zc7KWsYoeLt5E_P6hgERE60yuujJHXiWpuT9jFjj4gC367dihQ6w7V22qGaDAek1JHPJHKEsYT6vqwQAPm9GSln6px2RaJkIttMBpwOF398rKvWxlzAOmI4,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionException(
> ClientSessionFactoryImpl.java:425
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a425&c=E,1,BFL6YN5Kxy06vezMnsiL2nC8cdKvlemlxhvuHQ-FuWkoOzxoYzQuft1CeyeiQmEo-L40YbRsfc6dzmYWbWpQQb_xVyaLDbql7YwiFlA60a1xj1PSDmU,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$
> 2.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f2.run&c=E,1,sMyuWkYLp3LX6ibrSEXRcAezwbBM7qAOluPp_GC1jVesMs6jxDzxY4FFM3VjeigvvheAyp3Q7Oe5ACoqft2B-abCTUa6WF1bt7Uh4xjE6hHhWxo,&typo=1&ancr_add=1>
> (NettyConnector.java:1252
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a1252&c=E,1,3JJSPZ_KDZLXcsGld2w64FmGt132PiKafjM_G1ZXb-ulOdS6Dwv6XUv8QHJqHN6CKkClQ7lUkNnakzibZdks8vAdD3gLbxCOyjzJGO6K745CLD2v2F7UK5zJSQGG&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,bpAKAylKI6Drp540buFl0pkNNpNnX4xnMjY3zWwpNyF_392duOqGQPjpdfCPGxrpBvc7_eIyfJpxl4WuufWnNYLFKdIg0Nu9WDi6STLxtiY0tUhT7eeO2iA,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,CqAtDgvRoDEnuFjXXv-RSPQ-NcQ5-m4IRpyQ9oiN0rhM40sqLFbhY9BMbc0HKKA9gNrRlaFd_DPaSB11LNOv4Rdahsd5fJLOqXHsZnvg3ShSJw,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,C5wuD-LRtJ7605p6qERurgxxNLnHBKbC4B7VE9lfDl7f_JsOdDSkeiyMkYACWnhSckLFmjUW2eFKZawLMNYORn4gTtULbgzlBoD8LWUyNto,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,rMwMgnKWBZ0Z_t5VCFvBOA7J5McS_4eMRlTXeB6c0yMri_CRxIy8PDWJRjrRtRMI1QpqaQtuBEcG4sCoeTP06BI4NmwaneyL2iZ_EcqSA0sDvg,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,AmU9mOKxf3Tw98cD6iOKGEUsELr-mALW0vP5N2eC_TxCQIG4d3rxa4xf_W2vXyThIPw3Zmp6MJGpD2c95V3s3JefMOVZJGioR7UrvoKjJHS1Qr22gQ,,&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,uKLItaPZw48N90kGuOJ2giezmgZl49ttEaMZ2H4_Lr1mqtycK7JXSo7FWi90G25OhrIF9k-OxzV8LaDSYfMSQPd4uEksOAIsnz2-SSjTNFctPDccNQ,,&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,gIn498Y42-KQL16Kw7akU8ywkXBbJPMfxg7-Ivrxphmo6NJdJqsbcSdPEAdkwVYS3FgjewkEVfwxLXUtcBsU4QHUbsEaVo5060hl3qEBmZeRmUBspRNTPB0,&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,KRj2MYUCdG64pKABEvud93N21Qgh8ET58T7SDN4ZzK9ErKl92F_UyzJtq4y9eDIqY04euQg_cRGOTddiPcrePTpRRyy9U7AEk_WKzWUcU_VDwdQacLRCEHF4j6ag&typo=1&ancr_add=1>
> )
>
>
>
> -------------
>
> Log excerpt from AMQ docker “master” instance, showing lines missing from
> backup:
>
> 2024-01-11 22:38:54,844 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address DLQ supporting [ANYCAST]
>
> 2024-01-11 22:38:54,853 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
>
> 2024-01-11 22:38:54,905 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
>
> 2024-01-11 22:38:54,906 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
>
> 2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols
> [CORE,MQTT,AMQP,HORNETQ,STOMP,OPENWIRE]
>
> 2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server]
> AMQ221007: Server is now live
>
> 2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server]
> AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.31.2 [live,
> nodeID=320ff011-b0d2-11ee-a54c-0242ac110004]
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,WIQnILFOGT12V7T2V5JgdPTF8Y-yZZKUo4gSIgdk9Xb0RKhobbQh-PjrSOCrn9J79hRQkhxWG4u7HGnntjzO4n-TgjsgxAzGPARQB9OM7Zk,&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Wednesday, January 10, 2024 6:57 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > ...I’ll have to instrument the code for that, right? Or am I missing an
> easier way to generate those on-demand?
>
>
>
> I'm not clear on why you'd need to instrument the code to get thread
> dumps. You can generate thread dumps through a number of different
> commands, e.g.:
>
>
>
> - jstack
>
> - jcmd
>
> - kill -3
>
>
>
> You can execute these administratively or just set up a script that
> generates them every so often.
>
>
>
>
>
> Justin
>
>
>
> On Wed, Jan 10, 2024 at 7:43 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hey Justin,
>
> I did finally get a chance to run everything locally, and it all works
> smoothly with failovers in both directions and the AMQ-client recovers
> within 2-3 seconds and with a lot less log noise.
>
>
>
> I did not get the thread dumps yet because I’ll have to instrument the
> code for that, right? Or am I missing an easier way to generate those
> on-demand? This is happening in a k8s pod, so not easy to connect with
> jconsole.
>
>
>
> Thanks,
>
> Lino
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,P1-JTENl1pkpwfayJeVxTcbmyxoXqv5aWc8dgoVoQa2P9YJ6ITYanSwPPWI3OOvd-1ZuyvCfDdUgZSHNKSzXPNMT51ZJGff0MJI1JNYM4KnicxOXm0JmVYcKXz0s&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Wednesday, January 10, 2024 3:02 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > So our issue seems to be only for failover from live-to-backup,
> consistently.
>
>
>
> As far as I know the client doesn't make any distinction between these two
> cases so I'm guessing there's some kind of environmental difference that's
> making an impact.
>
>
>
> > Is there something that we are perhaps not doing on the client?
>
>
>
> I don't think so. The connection URL you provided previously looks good.
>
>
>
> > I will extend one of the example codes to use threads and sone of the
> AMQ-client wrapper code in an effort to isolate this behavior on a simpler
> code base, as you recommented.
>
>
>
> Before going down that road it might be better to inspect the ostensibly
> low-hanging fruit first. Did you happen to test everything locally? If so,
> what happened? Did you acquire any thread dumps during the delay as
> previously requested?
>
>
>
>
>
> Justin
>
>
>
> On Wed, Jan 10, 2024 at 3:04 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hey Justin,
>
>
>
> After starting fresh with version 2.31.2, the examples work out of the box
> for me as well.
>
> And, after ensuring that version 2.31.2 is running everywhere, client was
> still using 2.30.0, and refactoring to using the recommended shared-store
> HA, our AMQ-client is still behaving as before. That is, failover from
> live-to-backup, client recovery takes over 1 minute, and logs report what I
> posted earlier. However, failover from backup-to-live, client recovers
> within 5 seconds, which is the expected behavior, and our system is happy.
> Please see new log excerpts with more logging included for the 2 relevant
> threads.
>
>
>
> So our issue seems to be only for failover from live-to-backup,
> consistently. Is there something that we are perhaps not doing on the
> client?
>
>
>
> I will extend one of the example codes to use threads and sone of the
> AMQ-client wrapper code in an effort to isolate this behavior on a simpler
> code base, as you recommented.
>
>
>
> In the meantime, I thought I’d ping you in case you can think of something
> obvious that we might have missed, especially since we just moved from
> 2.30.0.
>
>
>
> Thanks again,
>
> Lino
>
>
>
> Our versions:
>
> ARTEMIS_VERSION=2.31.2
>
> JMX_VERSION=0.18.0
>
>
>
> Logs from failover from backup-to-live:
>
> 2024-01-10T19:52:29.588 [Thread-24 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-backup-svc/10.43.229.161:61617 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d7ff7885, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,O0LJ4VfRCemWEaMLULajHsgUo6YseAVDyBlrTm2G50vM20WhlTUV0ueY1DxMRAMStXAoc6wy5Xx3x1vi2XsnE5F7_iNbL-MZ03M4IS9G&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,VdcejKqF9B67eWPDv-PP_0gR6PEmUxFYHHtW8kyn3Gzg3q-HHzL-Lr7oztjj2LQumPJWO-1KF4OPavU8c3DW4zPgrUsidhnnCpoMUUn_&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,xKCDuAN84vJBjsCp75epCz7V1lR05TFhETdvpPcKntBTPnQk0zuuQSei7-ly1RPUUsHDP8ZAe-IevqwGtTGfEB_yNzw90QcOpEx5BaXOW6E,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,pq7BenEb7b9i-FYz1LnTy-gHYDdV-28FiSv3pkV_i-1TQZtRxakbT-rOKkgLgz2IDx875Dk5EE_OrwsUcew9TIsvrcZZZ9ZIlwboxMEq9xnR5odpQL8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,8mmrZ5alRiVlyEshqjlGpedpt7TuUVBJOy8DDm2qHLGoY6v3pR76llV59sMsYwy2v4rwLqiD9Rcf3Hg4QSCZo7rdO1rQgMNapCKji2jg7T4IJOk3OAeLnIs,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,RLTYzCoyknTJ1GB9szMq3Jg02QAnt7juedv1HCTYmmA4JU1DO6MlJhkvzOTBqHbeeOdKSvwkualBrhGt0fUk7Nl5YLzlgfepqTWcDiBmXQ-_xSMcAHPw&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,iYtEGEWoGf58Vq3CGAxBrH3iWZ1jWm1N1b3AOFPNlx0kCahcxHEwXT7ZAQ_h3ynXTeKJSkvmBiGUkbqE3bH7tGw-Gn8VV7R7LaCkxl9Dch5I4kQDfJDKSg,,&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,kH54RADAl8RcHDeCxJW9b-P4YJJFElj_Nbh6mHtmwlFdP2tKzzbK3wkAXg9tVqhIlsbd4gr2LJkn7DWZ4I-J8WVE-7UovVbVakuQTTxc2pbR9vuIDiEFztc,&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,V9-XUYgB7GoqPyGhEdE_CmtNood-yvXw6P9wRIqQOSYSWuTfiF1bhonQejBQfgJdHq8k2TsE1RAvcSOp_WzBE-ybl0l6rc5pIuMGdC1LUITpdk_Iuha7urSje4Y,&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,RNgnFH7dBkD27TX353Mc-kqneOLcqFXEgpGywxg8VduTTYNyYFt5VqkKfirozQw9V88WOI8XK4v9dvrtPIl-3u7cp673GvKxq3L-XgqypuDtSnkRIYnh813U&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-backup-svc/10.43.229.161:61617 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d43c72b6, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,L97BTn6AFLbUpYgHzAlQm9pDJ4Sn--TlGDpz05xOE3xd-qn9krfLJ6JzGTaiSpQwuVjuQ8KUPfUSW6N18pgw44-ay7eqYb2h0p3sxehZeXAC&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,PiedcDlDv-nxD-EF13k9j-sagD4clTOVvCs6xyDQvnrPzDUxc4xAbnHg0d7hnqeG4cVJZaO42LJvoNm59E3SmClgLAft-3N7F898HO6Obbkm&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,G5SZ7tGYpGBQzNRxdmIzsp4hWlUsmbY3_881k0N5lFoFBwuHt1dMCkKHI8wiZRGpS6qDY_UDBvDmKVQNRIHXIIyg5Df2nnWDCTLnbvWCO20fW0PC&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,MsagygznR7ie9qgJLlMogXq8ex-X4ST-fg_vbVbgcBanXur80DVykS1ftnzq0e4bWLE0k8EiNmlhlezemqTlBEncWMJz-AQAKdTxEk4WyVY,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,n7l9CUpZZbFH3ca7Iu2wxM3kNfged80mMRcu0Vn9wIoZz7aKzCP3c1OVR73iqF0oLbir7CjMHKAu3sfrR2cLs-EP7orqDWQKQApkzTZra9CJag,,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,PePvk640nUsvwJWNJY39J2R954rSgiFVPPSQLOsjYskMxtDnDVBuzQiZ8ut5SAPoqzV9uYW6GX3ekOAAoou9eu6VpHa_oFc2-7d27W00VBLU_o_Zv4wEUw,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,REmfZe55ytZRsqZvFkCNsYUcRO5Yd8PZwIlQfGystV0qUeF2mXSYaWKgleI098Ac40PND4ilF-ukqEWDY8N04AIRQgGgKGpG036xx7jNSc78YS6UbKWiMNUHfNc,&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,nnqfN76WKNs0ZYhVPzea17QO5aWkwmLqbfYrmQJ_qGQIgTzCL4PSXhjU9jy4mhbCUfl53cwF4ISYabr_zD1GUvCgv5m4dYW1_OROCBpe&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,vlJ5vYZAfrlVp9kL50a_jwWhqVW8-LMUgzeBb6mn-udcS_fMR1yld889bXgOXTMDiutFqKYX9XPh0RtxeXRLtnN31WqeOlH_CZv4_ZyhnOwXvm4J&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7Ba-VAkMm3x0E0f6F1f73TH_y9Ylo03hx9DEz4kqWzvrXy0gAv3D9zIM35eXIaB95UFxo72DkMpb0nKW_a-BIF89JobbRqjyVNLdC5K3L1EM520B&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:29.592 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:29.594 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:29.593 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:29.597 [Thread-6 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d4cf19d5
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.600 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 7b2392ac
>
> 2024-01-10T19:52:29.602 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.602 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.603 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = c1fb4c82
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.610 [Thread-8 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 06b7b3a3
>
> 2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:29.611 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 65680be4
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.624 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 97a109b7
>
> 2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.625 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.629 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 77b2de2f
>
> 2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
>
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:30.612 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = f92d7cd6
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.613 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 1d44895d
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.614 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = e8602acd
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:30.631 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 0a217bfd
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.632 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d2aa00d7
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.633 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 9a4f639e
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.634 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = ecdb1cbf
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:31.000 [Schedule-Runner]
> SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service
> as system
>
> 2024-01-10T19:52:32.001 [Schedule-Runner]
> SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service
> as system
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 2/200
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:32.116 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = a6b3d488
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.117 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = c0627cd9
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.118 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 71523eb3
>
> 2024-01-10T19:52:32.119 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.120 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:32.134 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 2/200
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:32.135 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 68efd0dc
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.137 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = daddef73
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.137 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 84813ce3
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.138 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 533990e9
>
> 2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
>
>
> 2024-01-10T19:52:34.370 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 3/200
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:34.371 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 1d8df71b
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:34.373 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = de20e193
>
> 2024-01-10T19:52:34.373 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 3/200
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:34.390 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:34.390 [Thread-10 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 4661fd5d
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:34.392 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d1c14a74
>
> 2024-01-10T19:52:34.392 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.404 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-10T19:52:34.405 [Thread-9 (ActiveMQ-client-netty-threads)]
> ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up
> backup config = TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc for live =
> TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.405 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 38, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.409 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-10T19:52:34.410 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 37, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.411 [Thread-24 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 10, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.411 [Thread-11 (ActiveMQ-client-netty-threads)]
> ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up
> backup config = TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc for live =
> TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.413 [Thread-24 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,S5999ok8Om6JjNfBLhC0R2hYdl3dGqPWsLpis0MTx38AhKXPNqMMlH41RrDz0UZsxZzSZbLvUEOYFEa7hwM4N7gAZfCS9udcqSmOwhLg&typo=1&ancr_add=1>
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,CF5ZPcR0aBH6uaRjLjSeWoBKLnRwUOOgvrD5MzWuerKGIs1FEa5vZF8rdF1kWSKbLwGQlm1IsOeyey0f1t7eiA1ApUgs1EDhL7ilSV0c6rFxEanucXI,&typo=1&ancr_add=1>,
> needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,nl9VlpvL5Lx7cxWUON3eMxxXV1my7fPq5WQLaWD1ozszhEPr1wVfqCwFilScj02MzRACloIhLZ8CgyMWoa_OqCu-Tf4folOdvKvtooRq&typo=1&ancr_add=1>
>
> 2024-01-10T19:52:34.415 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d7ff7885, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> ClientSessionFactoryImpl.java:410
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,awSP7U_0_MIm7Km9lwe59SrlpKsd9iUMA2y266CCeNZS6GJuEExdDYVS6EVMnlmPlVN0W1aqSx2Rss2Dm_YIM71s5j7OAUz6pAdGaTqnU_h2RyaoD6KyuoU,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> NettyConnector.java:835
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,8gAKfG0MZn0OMdhCHJ98evOezuGoCS0mNbGWplSwawxhF0ed3tPq9vKWoWUbn2xmoLJ34skkCTpI2tUuevSCeVNT00tNpFlOJlyAjGWu&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:666
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,tXm2hjddgUkJac0QpMg4nVt2gwg5e3ORdql8K6xsJlBZIxph-NuPt1Ks2iB0ZarzoFELPAuAerVMiQpEU3nkq1JiqLAju26hE6Y2c3MwmKRTj66O8tsfQ-WFSh4d&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,P0xsXGcezpL-KzlLSOt_rfr6b6RF51B2hthJzXMGD04QeyLyg6AnMvF4M8ZOEwF90ojRdT3CtOy0x9HBSpGK_mmF_6Zl7Muz2TKrZRLAxQRxhV5XnvYlwQ,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,qctWvKkM1LAyegiROEl3hsiEN3dPcBcH4jlkBConOP-G2cgGl6X6XG-kdeQuBBOC8BmtTf-VmlbpJTjC_wznn8I6g7pLOg1uof2v0j_r3yyK7U8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,KRtNFryEe6jhqdT3of2FZC9PF7zGlYTxJ3KPFwO6TViZkyaHiybwHP3JmOhQ4F7yZtorUVRe5phyCswDWLc8lIFBVowIJygjwxLmO-F3xJSonSQ,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,ltH-LxKKSiKev4cs4CgS53B1Sn1z2t9IkxY_vjCKeoXMalJfF0yTujWPogG1a9qkWV5DrAT_qCTk7MdQa3LpkAae5yHTe_19fREF16p6vMMJqGXqLjw,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,66enqLb5-TVz_cB0ZCkDW4SpX4c3v0jslINMsxhHSjXfBWrIFoM8PxfSS7_VWmc_MdfmjPHIKcLm4zfcCe4MggwEWLxB-RjiUMqnmc8ShDJLoit5&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,xM_aKOMjg39ORE7gTvaGKP72ytDNvt6tjanfNTzekduYVGBvn963oijBiSY1JOQeorfgcCf57lVZSYIQ20x8sOFAH5zNexuTI0rBtrkDrtcJpq-S&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,Jr0OyBaxbJX-r7nzGvHtUkNxDkYucNHeuCPW_Mqw5I4IS0dve_XCjayhv7uvFQTdinHTsHqf2ZtZAn_qz1ZnE04PEsE0KWGfHQUj6N5WIKcNhyQ,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,fTod1kMxyNrLe6wmpPVQB1vco-7IrNIs9Dx8h9kWYDCH7xYH8V5kj2iao_X4D62DX5S24Xi08LA-xy2zHwj-LIzSKOA38iBBAlHdMD7qu3lK&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,37VGZcLM1KBT3tttPbLrhK822id0s0HCTimMYb18Jk-8bCbWxvPl2N9_yFFLgXa5pC9DXbKX6c8DMHolUFme8KRWVQdBx_lbRwtpLIOp6O7uiuV0fttkzvI3jxdO&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,zIgfFXrrGxOB_lKTtscVc_jXQlq_pH7TMnaK0IkixM4MVqoSgc-Wp5VV-QGQWmUtW5dggcgtYpRa7OoRHCBvDew7fxYJq28eSZVguCiuBiH9DzdKTHAXC5o,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,b6YutPqRZaL3jtcBxfqj_SiU4M_x_Qx-GeSmBfL7c6CG24IUL7AdjJxpyosXZtCtKco4bSDRxOj7HJn5CWgbgbPzJ0TFuXzJjs7VRUKVhg,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,xdw7r1BqGJuJPUUyo2a9wj_y2KKTC-j8c0DAocpHKzpZZaVOgPcxdng0Xi_JMKvlGYmcUvlTfZMm2FYQ7pTrF20xWUreNaRQqtU6rlWB3kkuPmMNBALyTBs,&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,4iCAaiDF74c7cCKTdUIIlT6Qu6BmbQItK-gaAIrQtpNdie21ccXNS_dCydALepUtQNTgsVsVUs1NavHOugg0IlFK0CjTj8TKnThm55ffUeOx8zHc4oZUYpE_LRsz&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,qBGyQwtcrkAjVKJZ9cZzaW1ivhGQgGfhpcJUS_ww-vfcQVNfD9OjDwDX1XjpzJWYpo08MqSOrKLpgO8d72TBVtSGg2OQ4AmggCv3IUgKeA,,&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,Mm6fpsoFsFLUtanrkJ6cqlk2y1zCJbPDIfqxsd0pS_sfS05oTAHBNIXIUUd4iA8HJY0guikOejZmykugHqKeopxLvJE3UmpX9YX1Y3Gu2ToK-B8aRw,,&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:34.435 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 54, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.465 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 98, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.474 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 20, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.483 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 128, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.495 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 158, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.499 [Thread-4]
> AbstractProducerCreditsImpl.checkCredits:184 [] DEBUG - not asking for -1
> credits on activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,BYjREAStKM-0e-e5OOY86CxL1g9WrAP01m5ROYtsGCMHTh3wox28vU8_r7Y9e7QwRiDtJ065Xst_zM3u0oFx_K7Fr-ZkN1GBxPR6aWEKrm4nQA,,&typo=1&ancr_add=1>
>
> 2024-01-10T19:52:34.499 [Thread-4]
> AbstractProducerCreditsImpl.afterAcquired:98 [] DEBUG - AfterAcquire 333
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,r4o5lwz6oA1VbUpicHhTmEpC9U7BwWBMw8ZYMcrl4g-jVFykL22Cb2WJIZLOqv-hLbxRqgThYlmO40Jo84JsPHe-IYBjUU8WMQ-jy1lZqxq3YQk,&typo=1&ancr_add=1>,
> pendingCredits=65203
>
> 2024-01-10T19:52:34.506 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 106, performing as a failover operation now and recreating
> objects
>
>
>
> 2024-01-10T19:52:34.680 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 155, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.683 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 21, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.684 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 129, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.686 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 57, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.687 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 174, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.688 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 176, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.689 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 56, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.690 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 108, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.691 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 156, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 233, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 112, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.693 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 187, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.694 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 110, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.695 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 144, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.696 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 235, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.697 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 28, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 47, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 30, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.699 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 77, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.701 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 65, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.710 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 70, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 34, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 159, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.713 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 76, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.714 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 119, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.716 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 59, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.719 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 87, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.720 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 80, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 166, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 149, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.722 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 137, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.723 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 163, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.724 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 179, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.726 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 181, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 17, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 58, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.728 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 32, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 49, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 234, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.730 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 143, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.731 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 42, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 136, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 178, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.733 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 55, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.734 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 95, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.737 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 48, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 16, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 146, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.739 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 22, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.742 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 43, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.743 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 130, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.744 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 36, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.745 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 60, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.746 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 117, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.751 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 14, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.752 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 81, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.754 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 93, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 138, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 100, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.756 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 62, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.757 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 125, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.758 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 15, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 99, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 44, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.760 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 172, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.761 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 84, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 79, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 122, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.764 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 64, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.765 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 88, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.767 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 151, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.772 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.773 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 96, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.774 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 27, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.775 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 73, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.776 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 90, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.777 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 134, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.778 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 35, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.780 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 118, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 186, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 41, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.782 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 31, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 150, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 145, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.787 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 127, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.789 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 132, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 170, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 142, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.791 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 53, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.792 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 86, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.793 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 147, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 115, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 12, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.795 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 135, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.796 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 68, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.797 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 103, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.798 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 141, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 165, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 18, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.800 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 33, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.801 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 116, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.802 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 124, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 26, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 50, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.804 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 75, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 85, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 69, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 97, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 101, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.808 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 153, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.809 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 107, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.810 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 175, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.811 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 66, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.812 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 111, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 152, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 45, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.814 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 61, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.815 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 113, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.816 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 105, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 162, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 171, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.818 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 173, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 184, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.820 [Thread-15 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.afterAcquired:98 [aoy5vnfnjetz,6ek4wlg8c0ad]
> DEBUG - AfterAcquire 781 credits on address rpdm.v1.authentication,
> pendingCredits=63974
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.user.group.query
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.user.group.query, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.user.group.query
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.canvas.state.query
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.canvas.state.query, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.canvas.state.query
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:184 [z7wx0g93sdr7] DEBUG - not
> asking for -1 credits on acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.afterAcquired:98 [z7wx0g93sdr7] DEBUG -
> AfterAcquire 505 credits on address acf62840-864a-4eb4-a902-4861db94eb93,
> pendingCredits=65031
>
> 2024-01-10T19:52:34.824 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d43c72b6, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> ClientSessionFactoryImpl.java:410
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,dWlUyTFJQxRIqOG8poa5pxge5mYVJ48dUXfACv6LUtKsZzFSVGKyZagOScZPGCnB6cxsyzdobSyk8B6Bwd79504jcLncGDwqMeY-9K6dpOs,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> NettyConnector.java:835
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ZNE9b4nkKfo7M-_v-eXIZ86i5mBDuVpXhcuhH7d4Z3TXAYIWBA7iz2F7WJKgkGmWz9vMjcTKd3WNKgAqq_OrNuwfMP9rqwnZZdfDvOywCznI2b71&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:666
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,ud4sg1bGnGSsxR72jGNWKmRRRqC9RhLjc5Fa_mAH3joB4qQCFRRYETYRnfp_427BSgD3-id4mG56z7G-3qi9hl8xOglhqXamz4f_kpydTW4Drw,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,5ea91cHeAd9MIKFvy9sbzhVk4n_LaL3aA10YRDmCls04v12YLb2XMmXw6eBUcLVuIE_Z644AmEt_e0ljvQnrS_I6HavZOLh4zjfiHWUvesg,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,jDru8ObAZFnt6UJuRwWa5fbG9hi1-FGT2jjgjYcb0j9MnKP4NE8WTbKBHSHyKvKnv60VBWqMlES8saZuop8zduWr_U-bg4uKdFl24xth0Wc08-CvUBpd&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,FCoOt1eEzJLmEhVMnIIHAKa-kTFceMfQD6Kcwc4ZSxQI7u6ROHuAf16piW1PXfz5fwuSX1g87RD5aUJh-U4htAPPBkPWT1OqrJG8TvqS_LoMCTdPQPVU&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,RozNWdaPDnLkxzHk0PIIu7yOI0mq7J36SJtiYqHJSj83AaF6otFCOMT6S0SrKP0i5_HdKBbJFOfyhgDON-XzQcdSOvC2Z4N90Agiv9ZXimQSY87tDA,,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3xUOFzb6MCi6u1j2LRQL6CE5BWbu5eM6Ir7EsfF2oCaYUBtqnA1r3VDYEmgBiVBb7QypDxbOzY9knH1MLZZ3oikFpY_p69_553LHOTbV7g,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,9xoBaA3kGi2K8zQ8BuNLDq0dYmsnE1nuJw9sP_cDKKpUiUl1eHBvRBJF127g2yN9LNqLqZkyPVSq8YusAWP6k6MCI4_4n846oL1LY84Tow2q00oOukNL4Q,,&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,OqfGI8b5WUrx4fzs6Vhc7lLcb55MI-F2H4j9ic-9Fbs1_FhBfTLzdobmjbVxUHuNV9kcphYOQdlHoTf8bN7hu4yIeKsdhwLW0rasAMuEm4mWhNV8t2c,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,ml--e7XB5_TgtwWFh6lw23pXHYNiKpjFGKoUKmwrOwHyS3lnFQ78O6WUjq-OM1ITxXTrgD6P5K2E89TnqOgGcaOyn6aLaVIcvPhARMyQEqlFDI2mcnYwIfwADKB-&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,olHwwJjC2TUJmGxt3e1EpgRMfVMAAdmudKrqZ5ZByilJc51VAK_ZZHGL1mLqUqv59mtQqrIVOT0FHxcuVGnfmTsHrtP_Lfh6vEjgZmLr0Aa8EqAKSU0BDQ,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,3crdMkhGhlkrJXp-HsjZIEJOjIoajfvvI_g-uTOHgEc26-rm5spEwaDR4tTrIOKeN5HSQz3HfKcnanjUMfuquaMfvk7M7K2AtlUVxZdR9Q,,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,Qv7RvLuSjcsV2_72qHHLM6lmOI05RppEGGyR8280SVuKWQgOAl-EeFo8ZDeB_9Hw2f5ZgHbKDJbuB11rT-FBX6wf71ASJsIGNZYXzHn1HnNPy3pXaBaEgvrJvA,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,7hPEKeOrrW0p5qUGMytIu7avFNFhxNhXFzKgo6a-hDqm4ky01dphgxAxTpF5DmP3oUrVV1uWYwMbJuUFv1IoX_o9-H7Y6xXWdj067c1A&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,dU7lk32qJcXeW-cUtM1E65BB47JiLhSgJYTqhLIX373Pj0miueYi3kN7MYtAU5zT39qOtzzAiHcBVYWsGDB3N8R31seI_xbm-rQzfd0jzb6V1rSLy1kotiI,&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,BtR0iFYtXaxmAlygBDvC74BLAVELzJivsFTs_YS41zS_XagPPAr0UUKW1lNrafWE3WxaZcW_S5WbNn583_uHQ7aLAA9vskUJfxaMwUrM_DxAOd2N&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,nW_o_ayhGlEOytZThvhNsFjqKDq7oEaYAEjw_jVKEHAnXglYBlSGSA3yj7wyG-2DXRXiRZSD11UwsKx7fSGPuHkUUSoULIYqacVjSrfmUuEfG_A,&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:34.824 [Thread-19] JmsStaticConnectionPool.onException:78
> [] ERROR - Receive error occurred. javax.jms.JMSException:
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(
> ActiveMQConnection.java:714
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a714&c=E,1,3FhMyZ8H2FihvFzilmw5MqmH_tkqp5vHbCOJsN5x6sQNSLyFELuJt_Ej2M_24CH42Xx0iQesALGDJK8HHdR_PDaLploHa8R84-uwfftA3DUM&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(
> ActiveMQConnection.java:735
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a735&c=E,1,JvpZWdXDc673FDsIBWz8WgGu29y6j8H0zAczvf_UXYyUAzEMKtvXJ3tMkleaFsrHU-QdlAUi0bWddjnikuurWEEMfZs0jIDWZ3jnrx2__Vbg93yZ8oo,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(
> ClientSessionFactoryImpl.java:868
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a868&c=E,1,AznnL1cSXDTbCj6iQ4ovEeP71-oopbCcH0laxV9dB-Y_kBuerNfeOAXRkRLybAqgqQBjJZKBNFEoveCJjTGncMCguHnY0hvsdZuKIHz6BEhvRX7N&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(
> ClientSessionFactoryImpl.java:856
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a856&c=E,1,vBewrTh4rfp_ezwl572jqeiy0VBShHwQJNRT99yv4B54VM5Qaf2hTHrOq33emXwk_WSFrcueJSCKI90AxcN4jUznfP1ZLzL1dJdUz01zK4vN&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:802
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a802&c=E,1,EY-ry2DzdJloXWtaemW-dWnRy1-rr2mQFFjoCbmWjX6xHCNZF95ldY2zUgfhOhSQFfzm7eYV0PdRUTOwJrx9jCD4U-B5ztFqWmvqEmJQuqdCyQgypA,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,gHouFFMo7plejV1zi4ureD9QEDkW-WjvytRlH6DOyEt7oMO8ULkj6APov-gDhxtXAspTlk7qF6RYw-RrCqfSrXlcE0Cm45J61qJHt5Yc3zuBJJyEX8A,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,17MGSyeFWTwZKhb2DCqWeyFvdlinlKAunc4JSHJltjjiaCeT_1UjXFsTMtEiF_NYIOMI3brJbPkRW1QkJIhe_Br44jy1od8-BI3l37IfFJzZbcculrc,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,F6NyfiAiJI9fxHnOQnnpO7dLyoFavqwnNivvLL6x2LO54G8SpNhXjlUN748SkNWwe_oCD9JhI7u_BOk1uNbwSTYV0lXRyewBrwvvysSEOSCggiFCFRgjtA,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,TnFdrKl6nCZv9PR6lSsRZ2YN0r8LqpGQMRo3oHlgyE7_AgUMzrW12R4HeB5JhX8Akv-1iKeVGrnTLV2WeAbBOCM-fukhfRPsdUgq5pwe0yU,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,vAtPtyOuzaufOELKjMWnOBMrJEOShH0EzH9G3WL8gak4hWHmnVWPL1W7c_YnxRH2IerBQErg9VJApOiiX6TR6tcPau3xqRHrPyrsH70KHg2v&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,pak9EX0xrMkRTN71voUljZIOmUvksqdCCqWdwh8fq2R4OZXUQFIHNH3P9j0t-wpQStfSs_ZsT0a9BZYopaoyMSnMkN3xqvXn8A9dTehv&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,acJEflBFObRkFNBskLYk7AARfS1_CzZH3yFGYB9r_Ype7n0DAWduZhGpmImVc9k1pTnmFoA8L8UukMVvWBinxujU6r75ipmzh4I_cNQ5Gzmd97g5SAU4dRsf3Hw,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,aM5tVNMvOtLUxuoV8Dez1Y7fAS8KwME3peX3IqP_QPViIWgkZQahJW_1BMBQUo35u5h9hGJ3XSyDgkECwWOGDn6a9W-gctjh8WvUKx9yQ0sSLwheMcs5mhR2yVM,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,fUWcgLX3hupjGCjPYEByZ8xfzFx4iN-VnkLB9MarNu6OEOfrwnCkvJvtu6e-ZV-FREp0Ssisi_fezDu18eP9adS4mmcPPlJH69vnf0YDVg,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,KuAjbB9H9ddyi6unfXrVdNXylpB7l6qLIUfJARScyUnc7iH1GtvO0s1MAtB9fwkdqXq8idOJYZs_rg0iU-sN4aV24ID0Os5tsyvB34ba3ve2D3diROqixzkQuw4,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,EcYd8WT9GtMN9IXo58gsQpj5uNwcdCRtDfa0zG9t2jpvfdaSX5r9vLey2icJ-4wo3EyBMyUQzwyqYwC5BMfe4yHGjyo7h8jxmjcEKig4V0KyVa_SVq58NBT8&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,_a7OW7NorEoCm9eOnYjCi6TFl2__or_ickK-DwP4rrMmRX2Rj_TKaxpGjSAwCJZasllt1gefhShoob68MthMyp6bxv9N5fucG2HiF0g3yK4kqHTZjiFn7kxZ&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,-ttWHQzQDZ94E-HqKtYTKV2nuApkTk60Zy25iWFIykqfNe0o2seEul4UE_gETZr08B40bJjmZKMHq_-R0M8bQ8Fbu8twSkWx9MPLFUhc&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,624T4PuCyLmsS-UG0IoXRx3MMz4DqnU_1BuWfOz973WSpqfLsFFDAICu75KCoCwAs9O8WgKTEcBVqRhhf1JUqI3Znxem3LsG7Ml0Td3DnJJUoGYR0BE,&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7UL50c_ahB0IhSHbe9OlGABBKKE4OFyNM67SjmNrSo_LQhfViHAlLgvwdlRpbmDdymI5VaeOEklHjvlweLBz8Ske6wjc8J3CzM05VYenh6DzNiI,&typo=1&ancr_add=1>
> )
>
> Caused by: ActiveMQDisconnectedException[errorType=DISCONNECTED
> message=AMQ219015: The connection was disconnected because of server
> shutdown]
>
> ... 7 more
>
>
>
> 2024-01-10T19:52:34.828 [Thread-17 (ActiveMQ-client-global-threads)]
> ClientSessionImpl.acknowledge:1225 [] DEBUG - client ack messageID =
> 10054445
>
>
>
> 2024-01-10T19:52:34.862 [Thread-24 (ActiveMQ-client-global-threads)]
> JmsServerBase.logRequest:117 [] INFO - REQUEST IN:
>
>
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,01YVdf0IHIl4w2xIwUh5i7ToRiQfv6TxT-FRWSNW0n6e2aucHkWVyKxkYJTpG1jt7_EUCuQVS0NWGalDyf11AUIb3qVPj67IUtjJB484IdLKrHgbmSd-Q8Vg7AKZ&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Tuesday, January 9, 2024 1:57 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Both the "replicated-failback-static" and "replicated-failback" examples
> run without issue out of the box on my machine so I'm not sure what's going
> on with that. One thing that's odd is that out-of-the-box the line 122 [1]
> doesn't match the method call in your NPE. Perhaps you modified the example?
>
>
>
> Aside from that there were a handful of questions in my previous message
> you didn't address. I think the questions about thread dumps are the most
> pressing given that you indicated you're deleting the pod. My guess is that
> the client is getting hung in a network call that doesn't want to return
> due to the pod disappearing. A series of thread dumps during the minute
> delay would confirm or deny that.
>
>
>
>
>
> Justin
>
>
>
> [1]
> https://github.com/apache/activemq-artemis/blob/2.31.2/examples/features/ha/replicated-failback-static/src/main/java/org/apache/activemq/artemis/jms/example/ReplicatedFailbackStaticExample.java#L122
>
>
>
> On Tue, Jan 9, 2024 at 12:47 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hey Justing, thanks for the help.
>
>
>
> We are using version 2.31.2, which is pretty recent, if not the latest.
>
>
>
> With regards to the log traces, other AMQ threads show similar behavior,
> and our system isn’t doing much until AMQ-client stabilizes.
>
> I am running on a k3d cluster and the failover is triggered by me deleting
> the dm-activemq-live pod, which k8s then spins back up.
>
> Also, our slave config does NOT specify “allow-failback=true”, so the spin
> up of the new dm-activemq-live instance does not trigger a fail-back, which
> is intentional to eliminate failover-failback chaos in such a short time.
>
>
>
> I am trying to setup a simpler test case by using one of the examples,
> artemis/2.31.2/examples/features/ha/replicated-failback-static, however, I
> am getting this NPE on line 112:
>
> Caused by: java.lang.NullPointerException: Cannot invoke
> "javax.jms.TextMessage.getText()" because "message0" is null
>
>
>
> The NPE seems to indicate that, you cannot call consumer.receive() again
> even if NOT acknowledged, which seems to contradict the comments on the
> code. I have run into this same behavior in another example as well,
> artemis/2.31.2/examples/features/ha/replicated-failback.
>
>
>
> Can you please confirm the expected behavior of the example code in the
> 2.31.2 version? I can then modify the code accordingly. Or please point me
> to a working sample code.
>
>
>
> Thanks,
>
> Lino
>
>
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,lIOPl76F2R4EtenIAEJMeQ17kfmHQ6buC7G1W5TRefOcSfxvj_-3dLXV3MKDkCTkac_65Rw0SYmof66i91zGwgb_SS1nKaBpao_Lw7JNNk1mkE33&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Monday, January 8, 2024 9:20 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > Now, the backup process does start processing msgs right away...
>
>
>
> What evidence do you have of that?
>
>
>
> > ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute
> apart, before it stabilizes after the failover and is once again usable.
>
>
>
> The log snippets you provided show a reconnection attempt back to
> dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to
> dm-activemq-backup-svc:61617 both of which fail almost immediately. Then
> right after that another connection attempt is made to
> dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e.
> from 20:03:44.874 to 20:04:44.933). Are there any other logs during this
> minute? Did you happen to get any thread dumps during this time? If so,
> what did they show? If not, could you? What was happening with
> dm-activemq-live-svc during this time? Perhaps there was something weird
> going on with the network related to whatever you were doing with that
> machine to trigger failover.
>
>
>
> You might consider simplifying your test-case and then adding complexity
> bit by bit to see what triggers the difference in behavior. For example,
> run your test with the client and both brokers on the *same machine*. If
> that works then move just the client to a remote machine. If that works
> then test the client just reconnecting to the primary broker upon restart
> (i.e. no failover). Then finally put all three entities on separate
> machines and test failover. Hopefully such an exercise will give you a
> better understanding of where the problem might lie.
>
>
>
> Lastly, please indicate what version of ActiveMQ Artemis you're using.
>
>
>
>
>
> Justin
>
>
>
> On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hi Justin,
>
> I’m working with John Lilley on getting an ActiveMQ HA configuration
> operational. Our current HA config is replication with master and slave
> instances and with static connections, and it seems to be working, except
> that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect
> itself after a failover from the live to the backup instance. Now, the
> backup process does start processing msgs right away, but the
> ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart,
> before it stabilizes after the failover and is once again usable. I have
> included the relevant log traces from the Thread-0 ActiveMQ-client showing
> the Connection failure detection all the way through when the client is
> usable again, over 1 minute later. I am hoping that, based on the trace,
> you can find something that, perhaps, we should be doing differently in the
> client side?
>
>
>
> Here is our Client connection url:
>
> (tcp://dm-activemq-live-svc:
> 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
>
>
>
> Should we also be including the corresponding connector names?
>
>
>
> In an earlier email with John Lilley you recommended 2 brokers using
> “shared storage” instead of “replication”, which I will try next, however,
> since this issue seems to be on the client-side, I want to make sure our
> client-side is setup correctly.
>
>
>
> Thanks in advance for your help.
>
> Lino
>
>
>
> Thread-0 log traces:
>
> 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-live-svc/10.43.197.248:61616 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=a8a4a77c, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> ---
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and currentConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.197.248:61616
>
>
>
> 2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.11.76:61617
>
>
>
> 2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> ---
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and currentConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.197.248:61616
>
> 2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = ced14fbc
>
>
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.11.76:61617
>
>
>
> 2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
>
>
> 2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 10, performing as a failover operation now and recreating
> objects
>
>
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address
> activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,hqiYs6vnJzfq-X42G9W9ohJtYgn9LRPR-82RHxMyPnKUD5xBwCbo8W6b-PegfmAGwvEENlDO0k2IkoGFFegGxR-H6J72UHHlpEFbGtJzhCF3sucQ&typo=1&ancr_add=1>
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,KX9atZvl9XxpGPWobW9hXsmaLrVekfw2221zvD0PJ9Ik3fEQVyHKcZZCe7MQRciamxD-T-WDEk5Xa7_cAt0PSHwVW3jG7zzhq8XH9XIRC8EXmg,,&typo=1&ancr_add=1>,
> needed = 65536, arriving = 0
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,ZOCwWoQv34cYLuw9hZvJzBHfSrX6oQeQ_3-SetFMJa1hRC6lXgiKsAElOhfZ_vfnBhRGRJPZjSOqHIbwWO2ltSYYY7ElQjyItp0uDo79_jaDkw,,&typo=1&ancr_add=1>
>
> 2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=a8a4a77c, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> ClientSessionFactoryImpl.java:410
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,obbNTaFqn6mglgyZXy1S-0nBLNVtEYlkZBwAEKSGDGN0RuqlR_-7fIQPW4ELNqVFJdenAcKh65C3yQ2b_Lef7W01Tst6ZTrmI66KfHLae572jXv-HF3O&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> NettyConnector.java:835
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ytVVTXzfq-J1NXl2ct7T_47F5EbfZNQTXP8dlqw720EiCe32c6CLwlpk0-v4MzUD61FErq0nkRTalFpcTtIN8TDeBvz70rzb__TyeIDm48ZKfFU28qQ,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:666
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,KQYUuEgOaR6V8a_ZC_xBpKZmDneB4LTL1cQzn2Gqtla8ntHJ1fgRs9KelLgvbMd9zvBTg8maXUerYZYQZa-DlfK7fon0JZ0O74PNJ1SOCHqIPN4fZQ6Xv731OGOD&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,OSVh6NSORxz7qZ6eIuUDgWskoYfyXT4NNdjJYGMbH7HeWEUiymys2DnptjYKdmQfqjfHmI-egRZ31yWwd9MKzEd-gbbsuTUsqn_46niKaw,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,DM2boEBGg44zzcd0m37ITkOnfDrkqLE8ZSFGJpEI4AGo2GyOI2yKX08AEkKHxw_UnDMF1vccYEvESFDXjXBBjjIGO5TEsMLWZHbJF3AOIEQ73uc,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,wipgGeA7UVZIy1uuqvuxJEx4BA8TvYfSYmO617IgrvnOo-z7sn1vUjXFqIEU9M1HmedBVoHxB_LpvuEZ9p_biioPM8V-V9eS8i8oST46GvBhZZIc&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,0FNTjscT_d2CNIid0J8d_x7C7Peb6_neE-a_MZeKqkM1o_sPGujvpkXuiLv6kms9wSFyYqSux86C1MX2R8WKM8IZBsjKt7sCs03U2lRYlVbw2eTDg0ihSEV1nA,,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3yKPqxpIi9n43QJSCQiysKGlEYttZ5Dmuirc7jUAWNz6orLfIUCYwdDEJ3eU0QiHywsNAJdWqrvnx5rZK6-8TTJb6o8tr1WSvSL7omLF&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,6B4qLrj7q9DzDb0Trp9xgst7cOBhbD4hcEcGyPFul5E_mh3_oGmColB9YQkQTB1gJ6x5Mcm0VwvBLddpDpfFkcZwlzb4YNR9Ym5fcDclFW97pIrQ9oG4&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KqR93A_gEw1FpU7__PsYjSkOC5LH0zFTSKu5K9YlCdVAz3h_c9EkBndO7aliE4RSbK_qslEZp0b-YzJfCOxmB22jAfplsYxguMl1qjC0rWpNjfr9_fh4RjNL3AOK&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,mhChlKkihYjDw_Kk5SL1d_4eb7zhBlSzri_nwjOUsrs6MAvEfDAAjia07yTMUU6CQqc0WbDS2Fn8HJJI0UwJh1N7SKiz9dsP1gJLmgY66huLhXZE9c-3ECGXyIWQ&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,nn8IDmg6XQJLLXzH7kunXDNZa_3A7fUlnHdRCLoha7gQWJO1QeY32duGFAQMYaFHsfIbHbv142mei8HfCqRwrpuUfikYxLkFoXaD3pORZb9roBJs0H8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,DPLEvt-MPmmzv78LEkeXtYHM2q37m12TxX-nqjuIJurQdeXydsSakco9t_O7jgBbQ6y115xxdDb8IonFZbM86zgRDLcvUmeOSJuDmZybD_6JaTX0zpbQ&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,1vNFV-mDaLK92lJo0Mq6jvkCDzuwrSHT4lp1nB6KTzOCSO6comH5xNpEcvtZ_9pc0cQCD2kgDfB5xdxg0tAX2dF5oG5DHY6kxWXOBLnuAwiz1BkEFhw,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,aLydz0SGsclotqCro2upcTg44B_Ij7ckjApLu4B5JUwZKKdMo5z3RyUR4ikHFY8L3t_yB87Zp5atGRwoAPA1L9Mc9c681UjFrG16n-udVjeHtxQNxx8V-Nug&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1mZX-u-XHEZ_e4Ut7YETRvWm7U_qkC-q9QaPeHah4K4yNl5foyM1VSExCtYiisknkelDj6_K2MUHGc65Kd-6-JG_TXLby0OPWodJVjX3BtKaDNBUzusgpzT6ajL7&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-nbEPDxI7J-DFvbJfVVt2Cs1Vi_hN5bw-8vQF3SlM1nthw3VbJLWmS1h73cupUxqBoHCImTzDqI1m5gutx3ZriLpLpU7IqutHCqZuT6CTVWACPtj&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,-lUcdhDzaHWdAWLCc0IxWCFkKtKHxoEd64GWhzXAbeuG_Zm9mvn624BY6uS8MgQn-s2TMF8PbEPYVbOOy6BJuI8ByeiuAF9Y8f2I5wib&typo=1&ancr_add=1>
> )
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,T2UV-1dji1RODRcU4Ff4j_IzfTH2n47BatzyLZcEOU8VROFr3GS8re7CjDcnnHCSaPEtR-35J3z30dnYcz5K66rVsMVxgDiyh0FvEDKnN67naHjYbn0NpzAc&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
Recommendations for HA/shared-store using Azure FileShare for the PV or performant alternatives
Posted by Lino Pereira <li...@redpointglobal.com.INVALID>.
Hi Justin,
Upon testing our “shared-store” solution in our target Azure K8S cluster, we ran into performance issues, seemingly due to the use of an Azure FileShare for the shared-store PV. We are currently using "Azure FileShare NFS on Linux" and NIO for the journaling mode.
Azure FileShare info: https://learn.microsoft.com/en-us/azure/storage/files/files-nfs-protocol
What are your recommendations on choosing the best type of Azure FileShare to achieve the required performance?
Should we look at other journaling modes as a solution here?
Or, is going back to “replication”, with 2 backup instances to avoid split-brain, the best solution? Note that we do not need to persist messages for any longer than a couple of minutes.
Thanks,
Lino
[rg] <https://www.redpointglobal.com/>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Lino Pereira <li...@redpointglobal.com.INVALID>
Sent: Thursday, January 11, 2024 4:11 PM
To: users@activemq.apache.org
Subject: RE: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hey Justin,
So I am trying to verify our AMQ docker images in the local environment, basically our system runs as simple processes on my local Linux system, but the AMQ instances run in docker containers. Now, this works fine for standalone, and for the HA case, the “master” instance works fine with, or without, the “backup” instance running. However, the “backup” instance doesn’t seem to ever start up fully, that is, it is missing the AMQ221080, AMQ221003, AMQ221007 and AMQ221001 log lines, shown below from the “master” instance. As a result, it never announces itself in “Backup” mode as seen when run from k8s. I am using the same broker configs that run fine when AMQ instances are run natively (no docker).
Also, when I stop the “master” instance, the AMQ-client reports these exceptions and never connects. Any ideas on why this may be happening for the backup, which uses the same exact docker image, just with a different broker.xml?
Is there some fundamental thing that I am missing when running 2 AMQ instances from docker containers?
My goal is to validate that our AMQ docker images are functional outside of k8s.
With regards, to getting you the thread dumps, next on the chopping block, can I just attach them to the email, or is there an alternate path to get those to you? Apologize for the dumb questions, but I am new at this.
Thanks.
Lino
AMQ-client exceptions after killing the “master” instance:
2024-01-11T15:47:15.322 [Thread-0 (ActiveMQ-client-global-threads)] client.errorCreatingNettyConnection:741 [] ERROR - AMQ214016: Failed to create netty connection java.lang.IllegalStateException: No ActiveMQChannelHandler has been found while connecting to localhost/127.0.0.1:61617 from Channel with id = 0bec947c
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:980<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a980&c=E,1,9sVwvAhg-dCUoWlCTkknJyr35Kb1lORNDVxR6-uqw3mazBhZ5pG19VN92yx8WmUo86gmEfiM-fyes3vKIRRqmUr7xGxTepKfWp8f8YCxIKDeU3MjhJIbTwA,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:866<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a866&c=E,1,PDO7x8ScbX2lkRczVyc7Qx5qFYh8jYI_QZSxIyD15vNJbt-HUSyFIAsL54Aas_i1kj247doxU1D-TF1f6UNtgifP4lOW0jVh-Ca8bmu8X2BRDUZyZYrlZd4,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:848<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a848&c=E,1,M5S848Ei6ypG34y6nHCsxBxodJIiZs0WdncK5b_Paenx_yRNXYZToN-BG2GJ4M8V03LsOGQzIKSbFW7uKF8az4WwG4Xrzk0hIvgvf05YYBR-KraWq2Quc6cO&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(ClientSessionFactoryImpl.java:1212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1212&c=E,1,Zu7gFKlm4HmSOURefrcnPu7_omfp3z6NkhUulfWOlvCON5qFl8o_m__R5jhaoNXCjQUWg3afJOkgibeky8zBPzqugdJrnx3mDcRvktMSa8mHPzzyhu9E&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1333<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1333&c=E,1,QLuXIqCAvTMp9vzA3xlqCzu6uu_CAV6-6B3ZjST9QtAqqlHPPW031VktMnXrspcOD4IBb1JKvR2F8OzNZIi97M8NdBmUvk48uDHiFLvxVM4,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1253<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1253&c=E,1,HGg4Q1mxe7PWvtXuKXcO1vTG9pGtPTivn_BSZ-cDFd_RB5T6KlxMhp2Z2iKrHRFFrcmAJyClu3xTgdygcYT1cRvfTIU0mnOfoNn73hij&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(ClientSessionFactoryImpl.java:1496<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1496&c=E,1,YDsqwgPbq8QPr_ODsKRUE5-FJdCxEgI2EdJdlE_ogGX_Dtkdqu0ZUW5FnNQHKMwQfJEjscTg0sXGDaY7D27_1M-BNdURAJybIJ0mM63IpSP_Kix0E7dJj0xAAA,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1074<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1074&c=E,1,UaHGLwmlvYxEp5Zvya3OgHoGpFPLrtGO8817O18DQ_eCtU9hppVQz3ZSqMOZR4Ou0RyfDnJ25Dsho8qeOiDziqkIZ-8smasmw6AS-SC3C07T9A,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:959<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a959&c=E,1,S2vAOnIQrU8itB1DHvtgyBp5ozbblk23_CGr3q-ohZ6Nt35WRISFB1PbqeDDdjRyaj8N9tzZSNUm_TpeEwNQg143NiFeTG18rXYlnq4jakQe2_H0HkGTrg5Qow,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:697<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a697&c=E,1,HRBbQntC7regy9ruiCBdgTxHOjjHiqqEtdrKtyuFoRluNvypXTYG7MHAazXY4rr-7iE9ymvG-ccC_EuBMTBG5AjelMg7KfPvqiIaUkF1RfMIvoeiA8o,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,WwOwlRDdAEBeA1Gg0ArmZFo1it0EZxIt6RsmaWdNGx6wo_Lz4qKdMxIHy7U_9S5bgl-33cfGzNtlYG9YM9FZXucb7-SjGMsXWWRiyGWMdHRJ&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:559<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a559&c=E,1,Zc7KWsYoeLt5E_P6hgERE60yuujJHXiWpuT9jFjj4gC367dihQ6w7V22qGaDAek1JHPJHKEsYT6vqwQAPm9GSln6px2RaJkIttMBpwOF398rKvWxlzAOmI4,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionException(ClientSessionFactoryImpl.java:425<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a425&c=E,1,BFL6YN5Kxy06vezMnsiL2nC8cdKvlemlxhvuHQ-FuWkoOzxoYzQuft1CeyeiQmEo-L40YbRsfc6dzmYWbWpQQb_xVyaLDbql7YwiFlA60a1xj1PSDmU,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$2.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f2.run&c=E,1,sMyuWkYLp3LX6ibrSEXRcAezwbBM7qAOluPp_GC1jVesMs6jxDzxY4FFM3VjeigvvheAyp3Q7Oe5ACoqft2B-abCTUa6WF1bt7Uh4xjE6hHhWxo,&typo=1&ancr_add=1>(NettyConnector.java:1252<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a1252&c=E,1,3JJSPZ_KDZLXcsGld2w64FmGt132PiKafjM_G1ZXb-ulOdS6Dwv6XUv8QHJqHN6CKkClQ7lUkNnakzibZdks8vAdD3gLbxCOyjzJGO6K745CLD2v2F7UK5zJSQGG&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,bpAKAylKI6Drp540buFl0pkNNpNnX4xnMjY3zWwpNyF_392duOqGQPjpdfCPGxrpBvc7_eIyfJpxl4WuufWnNYLFKdIg0Nu9WDi6STLxtiY0tUhT7eeO2iA,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,CqAtDgvRoDEnuFjXXv-RSPQ-NcQ5-m4IRpyQ9oiN0rhM40sqLFbhY9BMbc0HKKA9gNrRlaFd_DPaSB11LNOv4Rdahsd5fJLOqXHsZnvg3ShSJw,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,C5wuD-LRtJ7605p6qERurgxxNLnHBKbC4B7VE9lfDl7f_JsOdDSkeiyMkYACWnhSckLFmjUW2eFKZawLMNYORn4gTtULbgzlBoD8LWUyNto,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,rMwMgnKWBZ0Z_t5VCFvBOA7J5McS_4eMRlTXeB6c0yMri_CRxIy8PDWJRjrRtRMI1QpqaQtuBEcG4sCoeTP06BI4NmwaneyL2iZ_EcqSA0sDvg,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,AmU9mOKxf3Tw98cD6iOKGEUsELr-mALW0vP5N2eC_TxCQIG4d3rxa4xf_W2vXyThIPw3Zmp6MJGpD2c95V3s3JefMOVZJGioR7UrvoKjJHS1Qr22gQ,,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,uKLItaPZw48N90kGuOJ2giezmgZl49ttEaMZ2H4_Lr1mqtycK7JXSo7FWi90G25OhrIF9k-OxzV8LaDSYfMSQPd4uEksOAIsnz2-SSjTNFctPDccNQ,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,gIn498Y42-KQL16Kw7akU8ywkXBbJPMfxg7-Ivrxphmo6NJdJqsbcSdPEAdkwVYS3FgjewkEVfwxLXUtcBsU4QHUbsEaVo5060hl3qEBmZeRmUBspRNTPB0,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,KRj2MYUCdG64pKABEvud93N21Qgh8ET58T7SDN4ZzK9ErKl92F_UyzJtq4y9eDIqY04euQg_cRGOTddiPcrePTpRRyy9U7AEk_WKzWUcU_VDwdQacLRCEHF4j6ag&typo=1&ancr_add=1>)
-------------
Log excerpt from AMQ docker “master” instance, showing lines missing from backup:
2024-01-11 22:38:54,844 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST]
2024-01-11 22:38:54,853 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2024-01-11 22:38:54,905 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2024-01-11 22:38:54,906 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols [CORE,MQTT,AMQP,HORNETQ,STOMP,OPENWIRE]
2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.31.2 [live, nodeID=320ff011-b0d2-11ee-a54c-0242ac110004]
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,WIQnILFOGT12V7T2V5JgdPTF8Y-yZZKUo4gSIgdk9Xb0RKhobbQh-PjrSOCrn9J79hRQkhxWG4u7HGnntjzO4n-TgjsgxAzGPARQB9OM7Zk,&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Wednesday, January 10, 2024 6:57 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> ...I’ll have to instrument the code for that, right? Or am I missing an easier way to generate those on-demand?
I'm not clear on why you'd need to instrument the code to get thread dumps. You can generate thread dumps through a number of different commands, e.g.:
- jstack
- jcmd
- kill -3
You can execute these administratively or just set up a script that generates them every so often.
Justin
On Wed, Jan 10, 2024 at 7:43 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hey Justin,
I did finally get a chance to run everything locally, and it all works smoothly with failovers in both directions and the AMQ-client recovers within 2-3 seconds and with a lot less log noise.
I did not get the thread dumps yet because I’ll have to instrument the code for that, right? Or am I missing an easier way to generate those on-demand? This is happening in a k8s pod, so not easy to connect with jconsole.
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,P1-JTENl1pkpwfayJeVxTcbmyxoXqv5aWc8dgoVoQa2P9YJ6ITYanSwPPWI3OOvd-1ZuyvCfDdUgZSHNKSzXPNMT51ZJGff0MJI1JNYM4KnicxOXm0JmVYcKXz0s&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Wednesday, January 10, 2024 3:02 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> So our issue seems to be only for failover from live-to-backup, consistently.
As far as I know the client doesn't make any distinction between these two cases so I'm guessing there's some kind of environmental difference that's making an impact.
> Is there something that we are perhaps not doing on the client?
I don't think so. The connection URL you provided previously looks good.
> I will extend one of the example codes to use threads and sone of the AMQ-client wrapper code in an effort to isolate this behavior on a simpler code base, as you recommented.
Before going down that road it might be better to inspect the ostensibly low-hanging fruit first. Did you happen to test everything locally? If so, what happened? Did you acquire any thread dumps during the delay as previously requested?
Justin
On Wed, Jan 10, 2024 at 3:04 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hey Justin,
After starting fresh with version 2.31.2, the examples work out of the box for me as well.
And, after ensuring that version 2.31.2 is running everywhere, client was still using 2.30.0, and refactoring to using the recommended shared-store HA, our AMQ-client is still behaving as before. That is, failover from live-to-backup, client recovery takes over 1 minute, and logs report what I posted earlier. However, failover from backup-to-live, client recovers within 5 seconds, which is the expected behavior, and our system is happy. Please see new log excerpts with more logging included for the 2 relevant threads.
So our issue seems to be only for failover from live-to-backup, consistently. Is there something that we are perhaps not doing on the client?
I will extend one of the example codes to use threads and sone of the AMQ-client wrapper code in an effort to isolate this behavior on a simpler code base, as you recommented.
In the meantime, I thought I’d ping you in case you can think of something obvious that we might have missed, especially since we just moved from 2.30.0.
Thanks again,
Lino
Our versions:
ARTEMIS_VERSION=2.31.2
JMX_VERSION=0.18.0
Logs from failover from backup-to-live:
2024-01-10T19:52:29.588 [Thread-24 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d7ff7885, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,O0LJ4VfRCemWEaMLULajHsgUo6YseAVDyBlrTm2G50vM20WhlTUV0ueY1DxMRAMStXAoc6wy5Xx3x1vi2XsnE5F7_iNbL-MZ03M4IS9G&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,VdcejKqF9B67eWPDv-PP_0gR6PEmUxFYHHtW8kyn3Gzg3q-HHzL-Lr7oztjj2LQumPJWO-1KF4OPavU8c3DW4zPgrUsidhnnCpoMUUn_&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,xKCDuAN84vJBjsCp75epCz7V1lR05TFhETdvpPcKntBTPnQk0zuuQSei7-ly1RPUUsHDP8ZAe-IevqwGtTGfEB_yNzw90QcOpEx5BaXOW6E,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,pq7BenEb7b9i-FYz1LnTy-gHYDdV-28FiSv3pkV_i-1TQZtRxakbT-rOKkgLgz2IDx875Dk5EE_OrwsUcew9TIsvrcZZZ9ZIlwboxMEq9xnR5odpQL8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,8mmrZ5alRiVlyEshqjlGpedpt7TuUVBJOy8DDm2qHLGoY6v3pR76llV59sMsYwy2v4rwLqiD9Rcf3Hg4QSCZo7rdO1rQgMNapCKji2jg7T4IJOk3OAeLnIs,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,RLTYzCoyknTJ1GB9szMq3Jg02QAnt7juedv1HCTYmmA4JU1DO6MlJhkvzOTBqHbeeOdKSvwkualBrhGt0fUk7Nl5YLzlgfepqTWcDiBmXQ-_xSMcAHPw&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,iYtEGEWoGf58Vq3CGAxBrH3iWZ1jWm1N1b3AOFPNlx0kCahcxHEwXT7ZAQ_h3ynXTeKJSkvmBiGUkbqE3bH7tGw-Gn8VV7R7LaCkxl9Dch5I4kQDfJDKSg,,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,kH54RADAl8RcHDeCxJW9b-P4YJJFElj_Nbh6mHtmwlFdP2tKzzbK3wkAXg9tVqhIlsbd4gr2LJkn7DWZ4I-J8WVE-7UovVbVakuQTTxc2pbR9vuIDiEFztc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,V9-XUYgB7GoqPyGhEdE_CmtNood-yvXw6P9wRIqQOSYSWuTfiF1bhonQejBQfgJdHq8k2TsE1RAvcSOp_WzBE-ybl0l6rc5pIuMGdC1LUITpdk_Iuha7urSje4Y,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,RNgnFH7dBkD27TX353Mc-kqneOLcqFXEgpGywxg8VduTTYNyYFt5VqkKfirozQw9V88WOI8XK4v9dvrtPIl-3u7cp673GvKxq3L-XgqypuDtSnkRIYnh813U&typo=1&ancr_add=1>)
2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d43c72b6, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,L97BTn6AFLbUpYgHzAlQm9pDJ4Sn--TlGDpz05xOE3xd-qn9krfLJ6JzGTaiSpQwuVjuQ8KUPfUSW6N18pgw44-ay7eqYb2h0p3sxehZeXAC&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,PiedcDlDv-nxD-EF13k9j-sagD4clTOVvCs6xyDQvnrPzDUxc4xAbnHg0d7hnqeG4cVJZaO42LJvoNm59E3SmClgLAft-3N7F898HO6Obbkm&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,G5SZ7tGYpGBQzNRxdmIzsp4hWlUsmbY3_881k0N5lFoFBwuHt1dMCkKHI8wiZRGpS6qDY_UDBvDmKVQNRIHXIIyg5Df2nnWDCTLnbvWCO20fW0PC&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,MsagygznR7ie9qgJLlMogXq8ex-X4ST-fg_vbVbgcBanXur80DVykS1ftnzq0e4bWLE0k8EiNmlhlezemqTlBEncWMJz-AQAKdTxEk4WyVY,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,n7l9CUpZZbFH3ca7Iu2wxM3kNfged80mMRcu0Vn9wIoZz7aKzCP3c1OVR73iqF0oLbir7CjMHKAu3sfrR2cLs-EP7orqDWQKQApkzTZra9CJag,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,PePvk640nUsvwJWNJY39J2R954rSgiFVPPSQLOsjYskMxtDnDVBuzQiZ8ut5SAPoqzV9uYW6GX3ekOAAoou9eu6VpHa_oFc2-7d27W00VBLU_o_Zv4wEUw,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,REmfZe55ytZRsqZvFkCNsYUcRO5Yd8PZwIlQfGystV0qUeF2mXSYaWKgleI098Ac40PND4ilF-ukqEWDY8N04AIRQgGgKGpG036xx7jNSc78YS6UbKWiMNUHfNc,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,nnqfN76WKNs0ZYhVPzea17QO5aWkwmLqbfYrmQJ_qGQIgTzCL4PSXhjU9jy4mhbCUfl53cwF4ISYabr_zD1GUvCgv5m4dYW1_OROCBpe&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,vlJ5vYZAfrlVp9kL50a_jwWhqVW8-LMUgzeBb6mn-udcS_fMR1yld889bXgOXTMDiutFqKYX9XPh0RtxeXRLtnN31WqeOlH_CZv4_ZyhnOwXvm4J&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7Ba-VAkMm3x0E0f6F1f73TH_y9Ylo03hx9DEz4kqWzvrXy0gAv3D9zIM35eXIaB95UFxo72DkMpb0nKW_a-BIF89JobbRqjyVNLdC5K3L1EM520B&typo=1&ancr_add=1>)
2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:29.592 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:29.594 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:29.593 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:29.597 [Thread-6 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d4cf19d5
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.600 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 7b2392ac
2024-01-10T19:52:29.602 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.602 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.603 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = c1fb4c82
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.610 [Thread-8 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 06b7b3a3
2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:29.611 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 65680be4
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.624 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 97a109b7
2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.625 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.629 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 77b2de2f
2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:30.612 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = f92d7cd6
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.613 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 1d44895d
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.614 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = e8602acd
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:30.631 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 0a217bfd
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.632 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d2aa00d7
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.633 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 9a4f639e
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.634 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = ecdb1cbf
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:31.000 [Schedule-Runner] SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service as system
2024-01-10T19:52:32.001 [Schedule-Runner] SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service as system
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 2/200
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:32.116 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = a6b3d488
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.117 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = c0627cd9
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.118 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 71523eb3
2024-01-10T19:52:32.119 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.120 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:32.134 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 2/200
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:32.135 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 68efd0dc
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.137 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = daddef73
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.137 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 84813ce3
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.138 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 533990e9
2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:34.370 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 3/200
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:34.371 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 1d8df71b
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:34.373 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = de20e193
2024-01-10T19:52:34.373 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 3/200
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:34.390 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:34.390 [Thread-10 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 4661fd5d
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:34.392 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d1c14a74
2024-01-10T19:52:34.392 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.404 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-10T19:52:34.405 [Thread-9 (ActiveMQ-client-netty-threads)] ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up backup config = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc for live = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.405 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 38, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.409 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-10T19:52:34.410 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 37, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.411 [Thread-24 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 10, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.411 [Thread-11 (ActiveMQ-client-netty-threads)] ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up backup config = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc for live = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.413 [Thread-24 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,S5999ok8Om6JjNfBLhC0R2hYdl3dGqPWsLpis0MTx38AhKXPNqMMlH41RrDz0UZsxZzSZbLvUEOYFEa7hwM4N7gAZfCS9udcqSmOwhLg&typo=1&ancr_add=1>
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,CF5ZPcR0aBH6uaRjLjSeWoBKLnRwUOOgvrD5MzWuerKGIs1FEa5vZF8rdF1kWSKbLwGQlm1IsOeyey0f1t7eiA1ApUgs1EDhL7ilSV0c6rFxEanucXI,&typo=1&ancr_add=1>, needed = 65536, arriving = 0
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,nl9VlpvL5Lx7cxWUON3eMxxXV1my7fPq5WQLaWD1ozszhEPr1wVfqCwFilScj02MzRACloIhLZ8CgyMWoa_OqCu-Tf4folOdvKvtooRq&typo=1&ancr_add=1>
2024-01-10T19:52:34.415 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d7ff7885, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,awSP7U_0_MIm7Km9lwe59SrlpKsd9iUMA2y266CCeNZS6GJuEExdDYVS6EVMnlmPlVN0W1aqSx2Rss2Dm_YIM71s5j7OAUz6pAdGaTqnU_h2RyaoD6KyuoU,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,8gAKfG0MZn0OMdhCHJ98evOezuGoCS0mNbGWplSwawxhF0ed3tPq9vKWoWUbn2xmoLJ34skkCTpI2tUuevSCeVNT00tNpFlOJlyAjGWu&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,tXm2hjddgUkJac0QpMg4nVt2gwg5e3ORdql8K6xsJlBZIxph-NuPt1Ks2iB0ZarzoFELPAuAerVMiQpEU3nkq1JiqLAju26hE6Y2c3MwmKRTj66O8tsfQ-WFSh4d&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,P0xsXGcezpL-KzlLSOt_rfr6b6RF51B2hthJzXMGD04QeyLyg6AnMvF4M8ZOEwF90ojRdT3CtOy0x9HBSpGK_mmF_6Zl7Muz2TKrZRLAxQRxhV5XnvYlwQ,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,qctWvKkM1LAyegiROEl3hsiEN3dPcBcH4jlkBConOP-G2cgGl6X6XG-kdeQuBBOC8BmtTf-VmlbpJTjC_wznn8I6g7pLOg1uof2v0j_r3yyK7U8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,KRtNFryEe6jhqdT3of2FZC9PF7zGlYTxJ3KPFwO6TViZkyaHiybwHP3JmOhQ4F7yZtorUVRe5phyCswDWLc8lIFBVowIJygjwxLmO-F3xJSonSQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,ltH-LxKKSiKev4cs4CgS53B1Sn1z2t9IkxY_vjCKeoXMalJfF0yTujWPogG1a9qkWV5DrAT_qCTk7MdQa3LpkAae5yHTe_19fREF16p6vMMJqGXqLjw,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,66enqLb5-TVz_cB0ZCkDW4SpX4c3v0jslINMsxhHSjXfBWrIFoM8PxfSS7_VWmc_MdfmjPHIKcLm4zfcCe4MggwEWLxB-RjiUMqnmc8ShDJLoit5&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,xM_aKOMjg39ORE7gTvaGKP72ytDNvt6tjanfNTzekduYVGBvn963oijBiSY1JOQeorfgcCf57lVZSYIQ20x8sOFAH5zNexuTI0rBtrkDrtcJpq-S&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,Jr0OyBaxbJX-r7nzGvHtUkNxDkYucNHeuCPW_Mqw5I4IS0dve_XCjayhv7uvFQTdinHTsHqf2ZtZAn_qz1ZnE04PEsE0KWGfHQUj6N5WIKcNhyQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,fTod1kMxyNrLe6wmpPVQB1vco-7IrNIs9Dx8h9kWYDCH7xYH8V5kj2iao_X4D62DX5S24Xi08LA-xy2zHwj-LIzSKOA38iBBAlHdMD7qu3lK&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,37VGZcLM1KBT3tttPbLrhK822id0s0HCTimMYb18Jk-8bCbWxvPl2N9_yFFLgXa5pC9DXbKX6c8DMHolUFme8KRWVQdBx_lbRwtpLIOp6O7uiuV0fttkzvI3jxdO&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,zIgfFXrrGxOB_lKTtscVc_jXQlq_pH7TMnaK0IkixM4MVqoSgc-Wp5VV-QGQWmUtW5dggcgtYpRa7OoRHCBvDew7fxYJq28eSZVguCiuBiH9DzdKTHAXC5o,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,b6YutPqRZaL3jtcBxfqj_SiU4M_x_Qx-GeSmBfL7c6CG24IUL7AdjJxpyosXZtCtKco4bSDRxOj7HJn5CWgbgbPzJ0TFuXzJjs7VRUKVhg,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,xdw7r1BqGJuJPUUyo2a9wj_y2KKTC-j8c0DAocpHKzpZZaVOgPcxdng0Xi_JMKvlGYmcUvlTfZMm2FYQ7pTrF20xWUreNaRQqtU6rlWB3kkuPmMNBALyTBs,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,4iCAaiDF74c7cCKTdUIIlT6Qu6BmbQItK-gaAIrQtpNdie21ccXNS_dCydALepUtQNTgsVsVUs1NavHOugg0IlFK0CjTj8TKnThm55ffUeOx8zHc4oZUYpE_LRsz&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,qBGyQwtcrkAjVKJZ9cZzaW1ivhGQgGfhpcJUS_ww-vfcQVNfD9OjDwDX1XjpzJWYpo08MqSOrKLpgO8d72TBVtSGg2OQ4AmggCv3IUgKeA,,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,Mm6fpsoFsFLUtanrkJ6cqlk2y1zCJbPDIfqxsd0pS_sfS05oTAHBNIXIUUd4iA8HJY0guikOejZmykugHqKeopxLvJE3UmpX9YX1Y3Gu2ToK-B8aRw,,&typo=1&ancr_add=1>)
2024-01-10T19:52:34.435 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 54, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.465 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 98, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.474 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 20, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.483 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 128, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.495 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 158, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.499 [Thread-4] AbstractProducerCreditsImpl.checkCredits:184 [] DEBUG - not asking for -1 credits on activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,BYjREAStKM-0e-e5OOY86CxL1g9WrAP01m5ROYtsGCMHTh3wox28vU8_r7Y9e7QwRiDtJ065Xst_zM3u0oFx_K7Fr-ZkN1GBxPR6aWEKrm4nQA,,&typo=1&ancr_add=1>
2024-01-10T19:52:34.499 [Thread-4] AbstractProducerCreditsImpl.afterAcquired:98 [] DEBUG - AfterAcquire 333 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,r4o5lwz6oA1VbUpicHhTmEpC9U7BwWBMw8ZYMcrl4g-jVFykL22Cb2WJIZLOqv-hLbxRqgThYlmO40Jo84JsPHe-IYBjUU8WMQ-jy1lZqxq3YQk,&typo=1&ancr_add=1>, pendingCredits=65203
2024-01-10T19:52:34.506 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 106, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.680 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 155, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.683 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 21, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.684 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 129, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.686 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 57, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.687 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 174, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.688 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 176, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.689 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 56, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.690 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 108, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.691 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 156, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 233, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 112, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.693 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 187, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.694 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 110, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.695 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 144, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.696 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 235, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.697 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 28, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 47, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 30, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.699 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 77, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.701 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 65, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.710 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 70, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 34, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 159, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.713 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 76, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.714 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 119, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.716 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 59, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.719 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 87, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.720 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 80, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 166, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 149, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.722 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 137, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.723 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 163, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.724 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 179, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.726 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 181, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 17, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 58, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.728 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 32, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 49, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 234, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.730 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 143, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.731 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 42, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 136, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 178, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.733 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 55, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.734 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 95, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.737 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 48, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 16, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 146, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.739 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 22, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.742 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 43, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.743 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 130, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.744 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 36, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.745 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 60, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.746 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 117, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.751 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 14, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.752 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 81, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.754 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 93, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 138, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 100, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.756 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 62, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.757 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 125, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.758 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 15, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 99, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 44, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.760 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 172, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.761 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 84, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 79, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 122, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.764 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 64, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.765 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 88, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.767 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 151, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.772 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.773 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 96, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.774 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 27, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.775 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 73, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.776 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 90, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.777 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 134, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.778 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 35, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.780 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 118, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 186, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 41, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.782 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 31, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 150, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 145, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.787 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 127, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.789 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 132, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 170, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 142, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.791 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 53, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.792 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 86, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.793 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 147, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 115, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 12, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.795 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 135, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.796 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 68, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.797 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 103, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.798 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 141, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 165, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 18, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.800 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 33, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.801 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 116, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.802 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 124, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 26, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 50, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.804 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 75, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 85, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 69, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 97, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 101, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.808 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 153, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.809 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 107, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.810 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 175, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.811 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 66, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.812 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 111, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 152, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 45, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.814 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 61, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.815 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 113, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.816 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 105, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 162, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 171, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.818 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 173, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 184, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.820 [Thread-15 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.afterAcquired:98 [aoy5vnfnjetz,6ek4wlg8c0ad] DEBUG - AfterAcquire 781 credits on address rpdm.v1.authentication, pendingCredits=63974
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.user.group.query
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.user.group.query, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.user.group.query
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.canvas.state.query
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.canvas.state.query, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.canvas.state.query
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:184 [z7wx0g93sdr7] DEBUG - not asking for -1 credits on acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.afterAcquired:98 [z7wx0g93sdr7] DEBUG - AfterAcquire 505 credits on address acf62840-864a-4eb4-a902-4861db94eb93, pendingCredits=65031
2024-01-10T19:52:34.824 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d43c72b6, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,dWlUyTFJQxRIqOG8poa5pxge5mYVJ48dUXfACv6LUtKsZzFSVGKyZagOScZPGCnB6cxsyzdobSyk8B6Bwd79504jcLncGDwqMeY-9K6dpOs,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ZNE9b4nkKfo7M-_v-eXIZ86i5mBDuVpXhcuhH7d4Z3TXAYIWBA7iz2F7WJKgkGmWz9vMjcTKd3WNKgAqq_OrNuwfMP9rqwnZZdfDvOywCznI2b71&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,ud4sg1bGnGSsxR72jGNWKmRRRqC9RhLjc5Fa_mAH3joB4qQCFRRYETYRnfp_427BSgD3-id4mG56z7G-3qi9hl8xOglhqXamz4f_kpydTW4Drw,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,5ea91cHeAd9MIKFvy9sbzhVk4n_LaL3aA10YRDmCls04v12YLb2XMmXw6eBUcLVuIE_Z644AmEt_e0ljvQnrS_I6HavZOLh4zjfiHWUvesg,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,jDru8ObAZFnt6UJuRwWa5fbG9hi1-FGT2jjgjYcb0j9MnKP4NE8WTbKBHSHyKvKnv60VBWqMlES8saZuop8zduWr_U-bg4uKdFl24xth0Wc08-CvUBpd&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,FCoOt1eEzJLmEhVMnIIHAKa-kTFceMfQD6Kcwc4ZSxQI7u6ROHuAf16piW1PXfz5fwuSX1g87RD5aUJh-U4htAPPBkPWT1OqrJG8TvqS_LoMCTdPQPVU&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,RozNWdaPDnLkxzHk0PIIu7yOI0mq7J36SJtiYqHJSj83AaF6otFCOMT6S0SrKP0i5_HdKBbJFOfyhgDON-XzQcdSOvC2Z4N90Agiv9ZXimQSY87tDA,,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3xUOFzb6MCi6u1j2LRQL6CE5BWbu5eM6Ir7EsfF2oCaYUBtqnA1r3VDYEmgBiVBb7QypDxbOzY9knH1MLZZ3oikFpY_p69_553LHOTbV7g,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,9xoBaA3kGi2K8zQ8BuNLDq0dYmsnE1nuJw9sP_cDKKpUiUl1eHBvRBJF127g2yN9LNqLqZkyPVSq8YusAWP6k6MCI4_4n846oL1LY84Tow2q00oOukNL4Q,,&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,OqfGI8b5WUrx4fzs6Vhc7lLcb55MI-F2H4j9ic-9Fbs1_FhBfTLzdobmjbVxUHuNV9kcphYOQdlHoTf8bN7hu4yIeKsdhwLW0rasAMuEm4mWhNV8t2c,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,ml--e7XB5_TgtwWFh6lw23pXHYNiKpjFGKoUKmwrOwHyS3lnFQ78O6WUjq-OM1ITxXTrgD6P5K2E89TnqOgGcaOyn6aLaVIcvPhARMyQEqlFDI2mcnYwIfwADKB-&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,olHwwJjC2TUJmGxt3e1EpgRMfVMAAdmudKrqZ5ZByilJc51VAK_ZZHGL1mLqUqv59mtQqrIVOT0FHxcuVGnfmTsHrtP_Lfh6vEjgZmLr0Aa8EqAKSU0BDQ,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,3crdMkhGhlkrJXp-HsjZIEJOjIoajfvvI_g-uTOHgEc26-rm5spEwaDR4tTrIOKeN5HSQz3HfKcnanjUMfuquaMfvk7M7K2AtlUVxZdR9Q,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,Qv7RvLuSjcsV2_72qHHLM6lmOI05RppEGGyR8280SVuKWQgOAl-EeFo8ZDeB_9Hw2f5ZgHbKDJbuB11rT-FBX6wf71ASJsIGNZYXzHn1HnNPy3pXaBaEgvrJvA,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,7hPEKeOrrW0p5qUGMytIu7avFNFhxNhXFzKgo6a-hDqm4ky01dphgxAxTpF5DmP3oUrVV1uWYwMbJuUFv1IoX_o9-H7Y6xXWdj067c1A&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,dU7lk32qJcXeW-cUtM1E65BB47JiLhSgJYTqhLIX373Pj0miueYi3kN7MYtAU5zT39qOtzzAiHcBVYWsGDB3N8R31seI_xbm-rQzfd0jzb6V1rSLy1kotiI,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,BtR0iFYtXaxmAlygBDvC74BLAVELzJivsFTs_YS41zS_XagPPAr0UUKW1lNrafWE3WxaZcW_S5WbNn583_uHQ7aLAA9vskUJfxaMwUrM_DxAOd2N&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,nW_o_ayhGlEOytZThvhNsFjqKDq7oEaYAEjw_jVKEHAnXglYBlSGSA3yj7wyG-2DXRXiRZSD11UwsKx7fSGPuHkUUSoULIYqacVjSrfmUuEfG_A,&typo=1&ancr_add=1>)
2024-01-10T19:52:34.824 [Thread-19] JmsStaticConnectionPool.onException:78 [] ERROR - Receive error occurred. javax.jms.JMSException: ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:714<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a714&c=E,1,3FhMyZ8H2FihvFzilmw5MqmH_tkqp5vHbCOJsN5x6sQNSLyFELuJt_Ej2M_24CH42Xx0iQesALGDJK8HHdR_PDaLploHa8R84-uwfftA3DUM&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:735<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a735&c=E,1,JvpZWdXDc673FDsIBWz8WgGu29y6j8H0zAczvf_UXYyUAzEMKtvXJ3tMkleaFsrHU-QdlAUi0bWddjnikuurWEEMfZs0jIDWZ3jnrx2__Vbg93yZ8oo,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:868<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a868&c=E,1,AznnL1cSXDTbCj6iQ4ovEeP71-oopbCcH0laxV9dB-Y_kBuerNfeOAXRkRLybAqgqQBjJZKBNFEoveCJjTGncMCguHnY0hvsdZuKIHz6BEhvRX7N&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:856<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a856&c=E,1,vBewrTh4rfp_ezwl572jqeiy0VBShHwQJNRT99yv4B54VM5Qaf2hTHrOq33emXwk_WSFrcueJSCKI90AxcN4jUznfP1ZLzL1dJdUz01zK4vN&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:802<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a802&c=E,1,EY-ry2DzdJloXWtaemW-dWnRy1-rr2mQFFjoCbmWjX6xHCNZF95ldY2zUgfhOhSQFfzm7eYV0PdRUTOwJrx9jCD4U-B5ztFqWmvqEmJQuqdCyQgypA,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,gHouFFMo7plejV1zi4ureD9QEDkW-WjvytRlH6DOyEt7oMO8ULkj6APov-gDhxtXAspTlk7qF6RYw-RrCqfSrXlcE0Cm45J61qJHt5Yc3zuBJJyEX8A,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,17MGSyeFWTwZKhb2DCqWeyFvdlinlKAunc4JSHJltjjiaCeT_1UjXFsTMtEiF_NYIOMI3brJbPkRW1QkJIhe_Br44jy1od8-BI3l37IfFJzZbcculrc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,F6NyfiAiJI9fxHnOQnnpO7dLyoFavqwnNivvLL6x2LO54G8SpNhXjlUN748SkNWwe_oCD9JhI7u_BOk1uNbwSTYV0lXRyewBrwvvysSEOSCggiFCFRgjtA,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,TnFdrKl6nCZv9PR6lSsRZ2YN0r8LqpGQMRo3oHlgyE7_AgUMzrW12R4HeB5JhX8Akv-1iKeVGrnTLV2WeAbBOCM-fukhfRPsdUgq5pwe0yU,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,vAtPtyOuzaufOELKjMWnOBMrJEOShH0EzH9G3WL8gak4hWHmnVWPL1W7c_YnxRH2IerBQErg9VJApOiiX6TR6tcPau3xqRHrPyrsH70KHg2v&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,pak9EX0xrMkRTN71voUljZIOmUvksqdCCqWdwh8fq2R4OZXUQFIHNH3P9j0t-wpQStfSs_ZsT0a9BZYopaoyMSnMkN3xqvXn8A9dTehv&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,acJEflBFObRkFNBskLYk7AARfS1_CzZH3yFGYB9r_Ype7n0DAWduZhGpmImVc9k1pTnmFoA8L8UukMVvWBinxujU6r75ipmzh4I_cNQ5Gzmd97g5SAU4dRsf3Hw,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,aM5tVNMvOtLUxuoV8Dez1Y7fAS8KwME3peX3IqP_QPViIWgkZQahJW_1BMBQUo35u5h9hGJ3XSyDgkECwWOGDn6a9W-gctjh8WvUKx9yQ0sSLwheMcs5mhR2yVM,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,fUWcgLX3hupjGCjPYEByZ8xfzFx4iN-VnkLB9MarNu6OEOfrwnCkvJvtu6e-ZV-FREp0Ssisi_fezDu18eP9adS4mmcPPlJH69vnf0YDVg,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,KuAjbB9H9ddyi6unfXrVdNXylpB7l6qLIUfJARScyUnc7iH1GtvO0s1MAtB9fwkdqXq8idOJYZs_rg0iU-sN4aV24ID0Os5tsyvB34ba3ve2D3diROqixzkQuw4,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,EcYd8WT9GtMN9IXo58gsQpj5uNwcdCRtDfa0zG9t2jpvfdaSX5r9vLey2icJ-4wo3EyBMyUQzwyqYwC5BMfe4yHGjyo7h8jxmjcEKig4V0KyVa_SVq58NBT8&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,_a7OW7NorEoCm9eOnYjCi6TFl2__or_ickK-DwP4rrMmRX2Rj_TKaxpGjSAwCJZasllt1gefhShoob68MthMyp6bxv9N5fucG2HiF0g3yK4kqHTZjiFn7kxZ&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,-ttWHQzQDZ94E-HqKtYTKV2nuApkTk60Zy25iWFIykqfNe0o2seEul4UE_gETZr08B40bJjmZKMHq_-R0M8bQ8Fbu8twSkWx9MPLFUhc&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,624T4PuCyLmsS-UG0IoXRx3MMz4DqnU_1BuWfOz973WSpqfLsFFDAICu75KCoCwAs9O8WgKTEcBVqRhhf1JUqI3Znxem3LsG7Ml0Td3DnJJUoGYR0BE,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7UL50c_ahB0IhSHbe9OlGABBKKE4OFyNM67SjmNrSo_LQhfViHAlLgvwdlRpbmDdymI5VaeOEklHjvlweLBz8Ske6wjc8J3CzM05VYenh6DzNiI,&typo=1&ancr_add=1>)
Caused by: ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
... 7 more
2024-01-10T19:52:34.828 [Thread-17 (ActiveMQ-client-global-threads)] ClientSessionImpl.acknowledge:1225 [] DEBUG - client ack messageID = 10054445
2024-01-10T19:52:34.862 [Thread-24 (ActiveMQ-client-global-threads)] JmsServerBase.logRequest:117 [] INFO - REQUEST IN:
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,01YVdf0IHIl4w2xIwUh5i7ToRiQfv6TxT-FRWSNW0n6e2aucHkWVyKxkYJTpG1jt7_EUCuQVS0NWGalDyf11AUIb3qVPj67IUtjJB484IdLKrHgbmSd-Q8Vg7AKZ&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Tuesday, January 9, 2024 1:57 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Both the "replicated-failback-static" and "replicated-failback" examples run without issue out of the box on my machine so I'm not sure what's going on with that. One thing that's odd is that out-of-the-box the line 122 [1] doesn't match the method call in your NPE. Perhaps you modified the example?
Aside from that there were a handful of questions in my previous message you didn't address. I think the questions about thread dumps are the most pressing given that you indicated you're deleting the pod. My guess is that the client is getting hung in a network call that doesn't want to return due to the pod disappearing. A series of thread dumps during the minute delay would confirm or deny that.
Justin
[1] https://github.com/apache/activemq-artemis/blob/2.31.2/examples/features/ha/replicated-failback-static/src/main/java/org/apache/activemq/artemis/jms/example/ReplicatedFailbackStaticExample.java#L122
On Tue, Jan 9, 2024 at 12:47 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hey Justing, thanks for the help.
We are using version 2.31.2, which is pretty recent, if not the latest.
With regards to the log traces, other AMQ threads show similar behavior, and our system isn’t doing much until AMQ-client stabilizes.
I am running on a k3d cluster and the failover is triggered by me deleting the dm-activemq-live pod, which k8s then spins back up.
Also, our slave config does NOT specify “allow-failback=true”, so the spin up of the new dm-activemq-live instance does not trigger a fail-back, which is intentional to eliminate failover-failback chaos in such a short time.
I am trying to setup a simpler test case by using one of the examples, artemis/2.31.2/examples/features/ha/replicated-failback-static, however, I am getting this NPE on line 112:
Caused by: java.lang.NullPointerException: Cannot invoke "javax.jms.TextMessage.getText()" because "message0" is null
The NPE seems to indicate that, you cannot call consumer.receive() again even if NOT acknowledged, which seems to contradict the comments on the code. I have run into this same behavior in another example as well, artemis/2.31.2/examples/features/ha/replicated-failback.
Can you please confirm the expected behavior of the example code in the 2.31.2 version? I can then modify the code accordingly. Or please point me to a working sample code.
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,lIOPl76F2R4EtenIAEJMeQ17kfmHQ6buC7G1W5TRefOcSfxvj_-3dLXV3MKDkCTkac_65Rw0SYmof66i91zGwgb_SS1nKaBpao_Lw7JNNk1mkE33&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Monday, January 8, 2024 9:20 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> Now, the backup process does start processing msgs right away...
What evidence do you have of that?
> ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable.
The log snippets you provided show a reconnection attempt back to dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to dm-activemq-backup-svc:61617 both of which fail almost immediately. Then right after that another connection attempt is made to dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e. from 20:03:44.874 to 20:04:44.933). Are there any other logs during this minute? Did you happen to get any thread dumps during this time? If so, what did they show? If not, could you? What was happening with dm-activemq-live-svc during this time? Perhaps there was something weird going on with the network related to whatever you were doing with that machine to trigger failover.
You might consider simplifying your test-case and then adding complexity bit by bit to see what triggers the difference in behavior. For example, run your test with the client and both brokers on the *same machine*. If that works then move just the client to a remote machine. If that works then test the client just reconnecting to the primary broker upon restart (i.e. no failover). Then finally put all three entities on separate machines and test failover. Hopefully such an exercise will give you a better understanding of where the problem might lie.
Lastly, please indicate what version of ActiveMQ Artemis you're using.
Justin
On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
I’m working with John Lilley on getting an ActiveMQ HA configuration operational. Our current HA config is replication with master and slave instances and with static connections, and it seems to be working, except that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect itself after a failover from the live to the backup instance. Now, the backup process does start processing msgs right away, but the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable. I have included the relevant log traces from the Thread-0 ActiveMQ-client showing the Connection failure detection all the way through when the client is usable again, over 1 minute later. I am hoping that, based on the trace, you can find something that, perhaps, we should be doing differently in the client side?
Here is our Client connection url:
(tcp://dm-activemq-live-svc: 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
Should we also be including the corresponding connector names?
In an earlier email with John Lilley you recommended 2 brokers using “shared storage” instead of “replication”, which I will try next, however, since this issue seems to be on the client-side, I want to make sure our client-side is setup correctly.
Thanks in advance for your help.
Lino
Thread-0 log traces:
2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
---
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616>
2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617<http://10.43.11.76:61617>
2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
---
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616>
2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = ced14fbc
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617<http://10.43.11.76:61617>
2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 10, performing as a failover operation now and recreating objects
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,hqiYs6vnJzfq-X42G9W9ohJtYgn9LRPR-82RHxMyPnKUD5xBwCbo8W6b-PegfmAGwvEENlDO0k2IkoGFFegGxR-H6J72UHHlpEFbGtJzhCF3sucQ&typo=1&ancr_add=1>
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,KX9atZvl9XxpGPWobW9hXsmaLrVekfw2221zvD0PJ9Ik3fEQVyHKcZZCe7MQRciamxD-T-WDEk5Xa7_cAt0PSHwVW3jG7zzhq8XH9XIRC8EXmg,,&typo=1&ancr_add=1>, needed = 65536, arriving = 0
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,ZOCwWoQv34cYLuw9hZvJzBHfSrX6oQeQ_3-SetFMJa1hRC6lXgiKsAElOhfZ_vfnBhRGRJPZjSOqHIbwWO2ltSYYY7ElQjyItp0uDo79_jaDkw,,&typo=1&ancr_add=1>
2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,obbNTaFqn6mglgyZXy1S-0nBLNVtEYlkZBwAEKSGDGN0RuqlR_-7fIQPW4ELNqVFJdenAcKh65C3yQ2b_Lef7W01Tst6ZTrmI66KfHLae572jXv-HF3O&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ytVVTXzfq-J1NXl2ct7T_47F5EbfZNQTXP8dlqw720EiCe32c6CLwlpk0-v4MzUD61FErq0nkRTalFpcTtIN8TDeBvz70rzb__TyeIDm48ZKfFU28qQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,KQYUuEgOaR6V8a_ZC_xBpKZmDneB4LTL1cQzn2Gqtla8ntHJ1fgRs9KelLgvbMd9zvBTg8maXUerYZYQZa-DlfK7fon0JZ0O74PNJ1SOCHqIPN4fZQ6Xv731OGOD&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,OSVh6NSORxz7qZ6eIuUDgWskoYfyXT4NNdjJYGMbH7HeWEUiymys2DnptjYKdmQfqjfHmI-egRZ31yWwd9MKzEd-gbbsuTUsqn_46niKaw,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,DM2boEBGg44zzcd0m37ITkOnfDrkqLE8ZSFGJpEI4AGo2GyOI2yKX08AEkKHxw_UnDMF1vccYEvESFDXjXBBjjIGO5TEsMLWZHbJF3AOIEQ73uc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,wipgGeA7UVZIy1uuqvuxJEx4BA8TvYfSYmO617IgrvnOo-z7sn1vUjXFqIEU9M1HmedBVoHxB_LpvuEZ9p_biioPM8V-V9eS8i8oST46GvBhZZIc&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,0FNTjscT_d2CNIid0J8d_x7C7Peb6_neE-a_MZeKqkM1o_sPGujvpkXuiLv6kms9wSFyYqSux86C1MX2R8WKM8IZBsjKt7sCs03U2lRYlVbw2eTDg0ihSEV1nA,,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3yKPqxpIi9n43QJSCQiysKGlEYttZ5Dmuirc7jUAWNz6orLfIUCYwdDEJ3eU0QiHywsNAJdWqrvnx5rZK6-8TTJb6o8tr1WSvSL7omLF&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,6B4qLrj7q9DzDb0Trp9xgst7cOBhbD4hcEcGyPFul5E_mh3_oGmColB9YQkQTB1gJ6x5Mcm0VwvBLddpDpfFkcZwlzb4YNR9Ym5fcDclFW97pIrQ9oG4&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KqR93A_gEw1FpU7__PsYjSkOC5LH0zFTSKu5K9YlCdVAz3h_c9EkBndO7aliE4RSbK_qslEZp0b-YzJfCOxmB22jAfplsYxguMl1qjC0rWpNjfr9_fh4RjNL3AOK&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,mhChlKkihYjDw_Kk5SL1d_4eb7zhBlSzri_nwjOUsrs6MAvEfDAAjia07yTMUU6CQqc0WbDS2Fn8HJJI0UwJh1N7SKiz9dsP1gJLmgY66huLhXZE9c-3ECGXyIWQ&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,nn8IDmg6XQJLLXzH7kunXDNZa_3A7fUlnHdRCLoha7gQWJO1QeY32duGFAQMYaFHsfIbHbv142mei8HfCqRwrpuUfikYxLkFoXaD3pORZb9roBJs0H8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,DPLEvt-MPmmzv78LEkeXtYHM2q37m12TxX-nqjuIJurQdeXydsSakco9t_O7jgBbQ6y115xxdDb8IonFZbM86zgRDLcvUmeOSJuDmZybD_6JaTX0zpbQ&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,1vNFV-mDaLK92lJo0Mq6jvkCDzuwrSHT4lp1nB6KTzOCSO6comH5xNpEcvtZ_9pc0cQCD2kgDfB5xdxg0tAX2dF5oG5DHY6kxWXOBLnuAwiz1BkEFhw,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,aLydz0SGsclotqCro2upcTg44B_Ij7ckjApLu4B5JUwZKKdMo5z3RyUR4ikHFY8L3t_yB87Zp5atGRwoAPA1L9Mc9c681UjFrG16n-udVjeHtxQNxx8V-Nug&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1mZX-u-XHEZ_e4Ut7YETRvWm7U_qkC-q9QaPeHah4K4yNl5foyM1VSExCtYiisknkelDj6_K2MUHGc65Kd-6-JG_TXLby0OPWodJVjX3BtKaDNBUzusgpzT6ajL7&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-nbEPDxI7J-DFvbJfVVt2Cs1Vi_hN5bw-8vQF3SlM1nthw3VbJLWmS1h73cupUxqBoHCImTzDqI1m5gutx3ZriLpLpU7IqutHCqZuT6CTVWACPtj&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,-lUcdhDzaHWdAWLCc0IxWCFkKtKHxoEd64GWhzXAbeuG_Zm9mvn624BY6uS8MgQn-s2TMF8PbEPYVbOOy6BJuI8ByeiuAF9Y8f2I5wib&typo=1&ancr_add=1>)
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,T2UV-1dji1RODRcU4Ff4j_IzfTH2n47BatzyLZcEOU8VROFr3GS8re7CjDcnnHCSaPEtR-35J3z30dnYcz5K66rVsMVxgDiyh0FvEDKnN67naHjYbn0NpzAc&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
Posted by Justin Bertram <jb...@apache.org>.
I'm glad to hear you've got things sorted out. Unfortunately I don't have
an explanation for you, but from the beginning this "smelled" like some
kind of environmental issue to me (which was the main reason I was so keen
on getting those thread dumps).
If you have any other questions don't hesitate to ask!
Justin
On Thu, Jan 11, 2024 at 6:34 PM Lino Pereira
<li...@redpointglobal.com.invalid> wrote:
> Justin, good news, be it somewhat inexplicable.
>
> I had to apply some updates to my Linux VM, which involved a reboot, and
> now, running our system on my k3d cluster, just like before, and failovers
> now work in both directions, instantly, and with no chatty exceptions from
> the AMQ-clients.
>
> I’ve done a bunch of failovers in both directions now, and they go
> completely unnoticed by the MAQ-client, smooth as silk.
>
> I am guessing my issues were due to something environmental in my system
> that got resolved either via the reboot, the update, or both. Perhaps you
> have a more scientific explanation?
>
> Anyways, I am both very happy that this saga is over, and a bit annoyed
> that I spent so much time on this, but, in retrospect, I did learn a lot
> about AMQ HA, so all is good.
>
>
>
> Thanks for all your help,
>
> Lino
>
>
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> Lino Pereira
>
> C++ Developer, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Lino Pereira <li...@redpointglobal.com.INVALID>
> *Sent:* Thursday, January 11, 2024 4:11 PM
> *To:* users@activemq.apache.org
> *Subject:* RE: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Hey Justin,
>
>
>
> So I am trying to verify our AMQ docker images in the local environment,
> basically our system runs as simple processes on my local Linux system, but
> the AMQ instances run in docker containers. Now, this works fine for
> standalone, and for the HA case, the “master” instance works fine with, or
> without, the “backup” instance running. However, the “backup” instance
> doesn’t seem to ever start up fully, that is, it is missing the AMQ221080,
> AMQ221003, AMQ221007 and AMQ221001 log lines, shown below from the
> “master” instance. As a result, it never announces itself in “Backup” mode
> as seen when run from k8s. I am using the same broker configs that run fine
> when AMQ instances are run natively (no docker).
>
>
>
> Also, when I stop the “master” instance, the AMQ-client reports these
> exceptions and never connects. Any ideas on why this may be happening for
> the backup, which uses the same exact docker image, just with a different
> broker.xml?
>
> Is there some fundamental thing that I am missing when running 2 AMQ
> instances from docker containers?
>
> My goal is to validate that our AMQ docker images are functional outside
> of k8s.
>
>
>
> With regards, to getting you the thread dumps, next on the chopping block,
> can I just attach them to the email, or is there an alternate path to get
> those to you? Apologize for the dumb questions, but I am new at this.
>
>
>
> Thanks.
>
> Lino
>
>
>
> AMQ-client exceptions after killing the “master” instance:
>
> 2024-01-11T15:47:15.322 [Thread-0 (ActiveMQ-client-global-threads)]
> client.errorCreatingNettyConnection:741 [] ERROR - AMQ214016: Failed to
> create netty connection java.lang.IllegalStateException: No
> ActiveMQChannelHandler has been found while connecting to localhost/
> 127.0.0.1:61617 from Channel with id = 0bec947c
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(
> NettyConnector.java:980
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a980&c=E,1,9sVwvAhg-dCUoWlCTkknJyr35Kb1lORNDVxR6-uqw3mazBhZ5pG19VN92yx8WmUo86gmEfiM-fyes3vKIRRqmUr7xGxTepKfWp8f8YCxIKDeU3MjhJIbTwA,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(
> NettyConnector.java:866
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a866&c=E,1,PDO7x8ScbX2lkRczVyc7Qx5qFYh8jYI_QZSxIyD15vNJbt-HUSyFIAsL54Aas_i1kj247doxU1D-TF1f6UNtgifP4lOW0jVh-Ca8bmu8X2BRDUZyZYrlZd4,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(
> NettyConnector.java:848
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a848&c=E,1,M5S848Ei6ypG34y6nHCsxBxodJIiZs0WdncK5b_Paenx_yRNXYZToN-BG2GJ4M8V03LsOGQzIKSbFW7uKF8az4WwG4Xrzk0hIvgvf05YYBR-KraWq2Quc6cO&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(
> ClientSessionFactoryImpl.java:1212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1212&c=E,1,Zu7gFKlm4HmSOURefrcnPu7_omfp3z6NkhUulfWOlvCON5qFl8o_m__R5jhaoNXCjQUWg3afJOkgibeky8zBPzqugdJrnx3mDcRvktMSa8mHPzzyhu9E&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(
> ClientSessionFactoryImpl.java:1333
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1333&c=E,1,QLuXIqCAvTMp9vzA3xlqCzu6uu_CAV6-6B3ZjST9QtAqqlHPPW031VktMnXrspcOD4IBb1JKvR2F8OzNZIi97M8NdBmUvk48uDHiFLvxVM4,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(
> ClientSessionFactoryImpl.java:1253
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1253&c=E,1,HGg4Q1mxe7PWvtXuKXcO1vTG9pGtPTivn_BSZ-cDFd_RB5T6KlxMhp2Z2iKrHRFFrcmAJyClu3xTgdygcYT1cRvfTIU0mnOfoNn73hij&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(
> ClientSessionFactoryImpl.java:1496
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1496&c=E,1,YDsqwgPbq8QPr_ODsKRUE5-FJdCxEgI2EdJdlE_ogGX_Dtkdqu0ZUW5FnNQHKMwQfJEjscTg0sXGDaY7D27_1M-BNdURAJybIJ0mM63IpSP_Kix0E7dJj0xAAA,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(
> ClientSessionFactoryImpl.java:1074
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1074&c=E,1,UaHGLwmlvYxEp5Zvya3OgHoGpFPLrtGO8817O18DQ_eCtU9hppVQz3ZSqMOZR4Ou0RyfDnJ25Dsho8qeOiDziqkIZ-8smasmw6AS-SC3C07T9A,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(
> ClientSessionFactoryImpl.java:959
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a959&c=E,1,S2vAOnIQrU8itB1DHvtgyBp5ozbblk23_CGr3q-ohZ6Nt35WRISFB1PbqeDDdjRyaj8N9tzZSNUm_TpeEwNQg143NiFeTG18rXYlnq4jakQe2_H0HkGTrg5Qow,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:697
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a697&c=E,1,HRBbQntC7regy9ruiCBdgTxHOjjHiqqEtdrKtyuFoRluNvypXTYG7MHAazXY4rr-7iE9ymvG-ccC_EuBMTBG5AjelMg7KfPvqiIaUkF1RfMIvoeiA8o,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,WwOwlRDdAEBeA1Gg0ArmZFo1it0EZxIt6RsmaWdNGx6wo_Lz4qKdMxIHy7U_9S5bgl-33cfGzNtlYG9YM9FZXucb7-SjGMsXWWRiyGWMdHRJ&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:559
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a559&c=E,1,Zc7KWsYoeLt5E_P6hgERE60yuujJHXiWpuT9jFjj4gC367dihQ6w7V22qGaDAek1JHPJHKEsYT6vqwQAPm9GSln6px2RaJkIttMBpwOF398rKvWxlzAOmI4,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionException(
> ClientSessionFactoryImpl.java:425
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a425&c=E,1,BFL6YN5Kxy06vezMnsiL2nC8cdKvlemlxhvuHQ-FuWkoOzxoYzQuft1CeyeiQmEo-L40YbRsfc6dzmYWbWpQQb_xVyaLDbql7YwiFlA60a1xj1PSDmU,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$
> 2.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f2.run&c=E,1,sMyuWkYLp3LX6ibrSEXRcAezwbBM7qAOluPp_GC1jVesMs6jxDzxY4FFM3VjeigvvheAyp3Q7Oe5ACoqft2B-abCTUa6WF1bt7Uh4xjE6hHhWxo,&typo=1&ancr_add=1>
> (NettyConnector.java:1252
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a1252&c=E,1,3JJSPZ_KDZLXcsGld2w64FmGt132PiKafjM_G1ZXb-ulOdS6Dwv6XUv8QHJqHN6CKkClQ7lUkNnakzibZdks8vAdD3gLbxCOyjzJGO6K745CLD2v2F7UK5zJSQGG&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,bpAKAylKI6Drp540buFl0pkNNpNnX4xnMjY3zWwpNyF_392duOqGQPjpdfCPGxrpBvc7_eIyfJpxl4WuufWnNYLFKdIg0Nu9WDi6STLxtiY0tUhT7eeO2iA,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,CqAtDgvRoDEnuFjXXv-RSPQ-NcQ5-m4IRpyQ9oiN0rhM40sqLFbhY9BMbc0HKKA9gNrRlaFd_DPaSB11LNOv4Rdahsd5fJLOqXHsZnvg3ShSJw,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,C5wuD-LRtJ7605p6qERurgxxNLnHBKbC4B7VE9lfDl7f_JsOdDSkeiyMkYACWnhSckLFmjUW2eFKZawLMNYORn4gTtULbgzlBoD8LWUyNto,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,rMwMgnKWBZ0Z_t5VCFvBOA7J5McS_4eMRlTXeB6c0yMri_CRxIy8PDWJRjrRtRMI1QpqaQtuBEcG4sCoeTP06BI4NmwaneyL2iZ_EcqSA0sDvg,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,AmU9mOKxf3Tw98cD6iOKGEUsELr-mALW0vP5N2eC_TxCQIG4d3rxa4xf_W2vXyThIPw3Zmp6MJGpD2c95V3s3JefMOVZJGioR7UrvoKjJHS1Qr22gQ,,&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,uKLItaPZw48N90kGuOJ2giezmgZl49ttEaMZ2H4_Lr1mqtycK7JXSo7FWi90G25OhrIF9k-OxzV8LaDSYfMSQPd4uEksOAIsnz2-SSjTNFctPDccNQ,,&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,gIn498Y42-KQL16Kw7akU8ywkXBbJPMfxg7-Ivrxphmo6NJdJqsbcSdPEAdkwVYS3FgjewkEVfwxLXUtcBsU4QHUbsEaVo5060hl3qEBmZeRmUBspRNTPB0,&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,KRj2MYUCdG64pKABEvud93N21Qgh8ET58T7SDN4ZzK9ErKl92F_UyzJtq4y9eDIqY04euQg_cRGOTddiPcrePTpRRyy9U7AEk_WKzWUcU_VDwdQacLRCEHF4j6ag&typo=1&ancr_add=1>
> )
>
>
>
> -------------
>
> Log excerpt from AMQ docker “master” instance, showing lines missing from
> backup:
>
> 2024-01-11 22:38:54,844 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address DLQ supporting [ANYCAST]
>
> 2024-01-11 22:38:54,853 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
>
> 2024-01-11 22:38:54,905 INFO [org.apache.activemq.artemis.core.server]
> AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
>
> 2024-01-11 22:38:54,906 INFO [org.apache.activemq.artemis.core.server]
> AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
>
> 2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server]
> AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols
> [CORE,MQTT,AMQP,HORNETQ,STOMP,OPENWIRE]
>
> 2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server]
> AMQ221007: Server is now live
>
> 2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server]
> AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.31.2 [live,
> nodeID=320ff011-b0d2-11ee-a54c-0242ac110004]
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,WIQnILFOGT12V7T2V5JgdPTF8Y-yZZKUo4gSIgdk9Xb0RKhobbQh-PjrSOCrn9J79hRQkhxWG4u7HGnntjzO4n-TgjsgxAzGPARQB9OM7Zk,&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Wednesday, January 10, 2024 6:57 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > ...I’ll have to instrument the code for that, right? Or am I missing an
> easier way to generate those on-demand?
>
>
>
> I'm not clear on why you'd need to instrument the code to get thread
> dumps. You can generate thread dumps through a number of different
> commands, e.g.:
>
>
>
> - jstack
>
> - jcmd
>
> - kill -3
>
>
>
> You can execute these administratively or just set up a script that
> generates them every so often.
>
>
>
>
>
> Justin
>
>
>
> On Wed, Jan 10, 2024 at 7:43 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hey Justin,
>
> I did finally get a chance to run everything locally, and it all works
> smoothly with failovers in both directions and the AMQ-client recovers
> within 2-3 seconds and with a lot less log noise.
>
>
>
> I did not get the thread dumps yet because I’ll have to instrument the
> code for that, right? Or am I missing an easier way to generate those
> on-demand? This is happening in a k8s pod, so not easy to connect with
> jconsole.
>
>
>
> Thanks,
>
> Lino
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,P1-JTENl1pkpwfayJeVxTcbmyxoXqv5aWc8dgoVoQa2P9YJ6ITYanSwPPWI3OOvd-1ZuyvCfDdUgZSHNKSzXPNMT51ZJGff0MJI1JNYM4KnicxOXm0JmVYcKXz0s&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Wednesday, January 10, 2024 3:02 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > So our issue seems to be only for failover from live-to-backup,
> consistently.
>
>
>
> As far as I know the client doesn't make any distinction between these two
> cases so I'm guessing there's some kind of environmental difference that's
> making an impact.
>
>
>
> > Is there something that we are perhaps not doing on the client?
>
>
>
> I don't think so. The connection URL you provided previously looks good.
>
>
>
> > I will extend one of the example codes to use threads and sone of the
> AMQ-client wrapper code in an effort to isolate this behavior on a simpler
> code base, as you recommented.
>
>
>
> Before going down that road it might be better to inspect the ostensibly
> low-hanging fruit first. Did you happen to test everything locally? If so,
> what happened? Did you acquire any thread dumps during the delay as
> previously requested?
>
>
>
>
>
> Justin
>
>
>
> On Wed, Jan 10, 2024 at 3:04 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hey Justin,
>
>
>
> After starting fresh with version 2.31.2, the examples work out of the box
> for me as well.
>
> And, after ensuring that version 2.31.2 is running everywhere, client was
> still using 2.30.0, and refactoring to using the recommended shared-store
> HA, our AMQ-client is still behaving as before. That is, failover from
> live-to-backup, client recovery takes over 1 minute, and logs report what I
> posted earlier. However, failover from backup-to-live, client recovers
> within 5 seconds, which is the expected behavior, and our system is happy.
> Please see new log excerpts with more logging included for the 2 relevant
> threads.
>
>
>
> So our issue seems to be only for failover from live-to-backup,
> consistently. Is there something that we are perhaps not doing on the
> client?
>
>
>
> I will extend one of the example codes to use threads and sone of the
> AMQ-client wrapper code in an effort to isolate this behavior on a simpler
> code base, as you recommented.
>
>
>
> In the meantime, I thought I’d ping you in case you can think of something
> obvious that we might have missed, especially since we just moved from
> 2.30.0.
>
>
>
> Thanks again,
>
> Lino
>
>
>
> Our versions:
>
> ARTEMIS_VERSION=2.31.2
>
> JMX_VERSION=0.18.0
>
>
>
> Logs from failover from backup-to-live:
>
> 2024-01-10T19:52:29.588 [Thread-24 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-backup-svc/10.43.229.161:61617 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d7ff7885, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,O0LJ4VfRCemWEaMLULajHsgUo6YseAVDyBlrTm2G50vM20WhlTUV0ueY1DxMRAMStXAoc6wy5Xx3x1vi2XsnE5F7_iNbL-MZ03M4IS9G&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,VdcejKqF9B67eWPDv-PP_0gR6PEmUxFYHHtW8kyn3Gzg3q-HHzL-Lr7oztjj2LQumPJWO-1KF4OPavU8c3DW4zPgrUsidhnnCpoMUUn_&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,xKCDuAN84vJBjsCp75epCz7V1lR05TFhETdvpPcKntBTPnQk0zuuQSei7-ly1RPUUsHDP8ZAe-IevqwGtTGfEB_yNzw90QcOpEx5BaXOW6E,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,pq7BenEb7b9i-FYz1LnTy-gHYDdV-28FiSv3pkV_i-1TQZtRxakbT-rOKkgLgz2IDx875Dk5EE_OrwsUcew9TIsvrcZZZ9ZIlwboxMEq9xnR5odpQL8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,8mmrZ5alRiVlyEshqjlGpedpt7TuUVBJOy8DDm2qHLGoY6v3pR76llV59sMsYwy2v4rwLqiD9Rcf3Hg4QSCZo7rdO1rQgMNapCKji2jg7T4IJOk3OAeLnIs,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,RLTYzCoyknTJ1GB9szMq3Jg02QAnt7juedv1HCTYmmA4JU1DO6MlJhkvzOTBqHbeeOdKSvwkualBrhGt0fUk7Nl5YLzlgfepqTWcDiBmXQ-_xSMcAHPw&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,iYtEGEWoGf58Vq3CGAxBrH3iWZ1jWm1N1b3AOFPNlx0kCahcxHEwXT7ZAQ_h3ynXTeKJSkvmBiGUkbqE3bH7tGw-Gn8VV7R7LaCkxl9Dch5I4kQDfJDKSg,,&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,kH54RADAl8RcHDeCxJW9b-P4YJJFElj_Nbh6mHtmwlFdP2tKzzbK3wkAXg9tVqhIlsbd4gr2LJkn7DWZ4I-J8WVE-7UovVbVakuQTTxc2pbR9vuIDiEFztc,&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,V9-XUYgB7GoqPyGhEdE_CmtNood-yvXw6P9wRIqQOSYSWuTfiF1bhonQejBQfgJdHq8k2TsE1RAvcSOp_WzBE-ybl0l6rc5pIuMGdC1LUITpdk_Iuha7urSje4Y,&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,RNgnFH7dBkD27TX353Mc-kqneOLcqFXEgpGywxg8VduTTYNyYFt5VqkKfirozQw9V88WOI8XK4v9dvrtPIl-3u7cp673GvKxq3L-XgqypuDtSnkRIYnh813U&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-backup-svc/10.43.229.161:61617 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d43c72b6, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,L97BTn6AFLbUpYgHzAlQm9pDJ4Sn--TlGDpz05xOE3xd-qn9krfLJ6JzGTaiSpQwuVjuQ8KUPfUSW6N18pgw44-ay7eqYb2h0p3sxehZeXAC&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,PiedcDlDv-nxD-EF13k9j-sagD4clTOVvCs6xyDQvnrPzDUxc4xAbnHg0d7hnqeG4cVJZaO42LJvoNm59E3SmClgLAft-3N7F898HO6Obbkm&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,G5SZ7tGYpGBQzNRxdmIzsp4hWlUsmbY3_881k0N5lFoFBwuHt1dMCkKHI8wiZRGpS6qDY_UDBvDmKVQNRIHXIIyg5Df2nnWDCTLnbvWCO20fW0PC&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,MsagygznR7ie9qgJLlMogXq8ex-X4ST-fg_vbVbgcBanXur80DVykS1ftnzq0e4bWLE0k8EiNmlhlezemqTlBEncWMJz-AQAKdTxEk4WyVY,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,n7l9CUpZZbFH3ca7Iu2wxM3kNfged80mMRcu0Vn9wIoZz7aKzCP3c1OVR73iqF0oLbir7CjMHKAu3sfrR2cLs-EP7orqDWQKQApkzTZra9CJag,,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,PePvk640nUsvwJWNJY39J2R954rSgiFVPPSQLOsjYskMxtDnDVBuzQiZ8ut5SAPoqzV9uYW6GX3ekOAAoou9eu6VpHa_oFc2-7d27W00VBLU_o_Zv4wEUw,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,REmfZe55ytZRsqZvFkCNsYUcRO5Yd8PZwIlQfGystV0qUeF2mXSYaWKgleI098Ac40PND4ilF-ukqEWDY8N04AIRQgGgKGpG036xx7jNSc78YS6UbKWiMNUHfNc,&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,nnqfN76WKNs0ZYhVPzea17QO5aWkwmLqbfYrmQJ_qGQIgTzCL4PSXhjU9jy4mhbCUfl53cwF4ISYabr_zD1GUvCgv5m4dYW1_OROCBpe&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,vlJ5vYZAfrlVp9kL50a_jwWhqVW8-LMUgzeBb6mn-udcS_fMR1yld889bXgOXTMDiutFqKYX9XPh0RtxeXRLtnN31WqeOlH_CZv4_ZyhnOwXvm4J&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7Ba-VAkMm3x0E0f6F1f73TH_y9Ylo03hx9DEz4kqWzvrXy0gAv3D9zIM35eXIaB95UFxo72DkMpb0nKW_a-BIF89JobbRqjyVNLdC5K3L1EM520B&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:29.592 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:29.594 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:29.593 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:29.597 [Thread-6 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d4cf19d5
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.600 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 7b2392ac
>
> 2024-01-10T19:52:29.602 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.602 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.603 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = c1fb4c82
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.610 [Thread-8 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 06b7b3a3
>
> 2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:29.611 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 65680be4
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.624 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 97a109b7
>
> 2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.625 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.629 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 77b2de2f
>
> 2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
>
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:30.612 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = f92d7cd6
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.613 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 1d44895d
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.614 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = e8602acd
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:30.631 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 0a217bfd
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.632 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d2aa00d7
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.633 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 9a4f639e
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.634 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = ecdb1cbf
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:31.000 [Schedule-Runner]
> SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service
> as system
>
> 2024-01-10T19:52:32.001 [Schedule-Runner]
> SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service
> as system
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 2/200
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:32.116 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = a6b3d488
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.117 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = c0627cd9
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.118 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 71523eb3
>
> 2024-01-10T19:52:32.119 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.120 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:32.134 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 2/200
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:32.135 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 68efd0dc
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.137 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = daddef73
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.137 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 84813ce3
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.138 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 533990e9
>
> 2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
>
>
> 2024-01-10T19:52:34.370 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 3/200
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:34.371 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 1d8df71b
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:34.373 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = de20e193
>
> 2024-01-10T19:52:34.373 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 3/200
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:34.390 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:34.390 [Thread-10 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 4661fd5d
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:34.392 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d1c14a74
>
> 2024-01-10T19:52:34.392 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.404 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-10T19:52:34.405 [Thread-9 (ActiveMQ-client-netty-threads)]
> ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up
> backup config = TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc for live =
> TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.405 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 38, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.409 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-10T19:52:34.410 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 37, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.411 [Thread-24 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 10, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.411 [Thread-11 (ActiveMQ-client-netty-threads)]
> ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up
> backup config = TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc for live =
> TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.413 [Thread-24 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,S5999ok8Om6JjNfBLhC0R2hYdl3dGqPWsLpis0MTx38AhKXPNqMMlH41RrDz0UZsxZzSZbLvUEOYFEa7hwM4N7gAZfCS9udcqSmOwhLg&typo=1&ancr_add=1>
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,CF5ZPcR0aBH6uaRjLjSeWoBKLnRwUOOgvrD5MzWuerKGIs1FEa5vZF8rdF1kWSKbLwGQlm1IsOeyey0f1t7eiA1ApUgs1EDhL7ilSV0c6rFxEanucXI,&typo=1&ancr_add=1>,
> needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,nl9VlpvL5Lx7cxWUON3eMxxXV1my7fPq5WQLaWD1ozszhEPr1wVfqCwFilScj02MzRACloIhLZ8CgyMWoa_OqCu-Tf4folOdvKvtooRq&typo=1&ancr_add=1>
>
> 2024-01-10T19:52:34.415 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d7ff7885, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> ClientSessionFactoryImpl.java:410
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,awSP7U_0_MIm7Km9lwe59SrlpKsd9iUMA2y266CCeNZS6GJuEExdDYVS6EVMnlmPlVN0W1aqSx2Rss2Dm_YIM71s5j7OAUz6pAdGaTqnU_h2RyaoD6KyuoU,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> NettyConnector.java:835
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,8gAKfG0MZn0OMdhCHJ98evOezuGoCS0mNbGWplSwawxhF0ed3tPq9vKWoWUbn2xmoLJ34skkCTpI2tUuevSCeVNT00tNpFlOJlyAjGWu&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:666
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,tXm2hjddgUkJac0QpMg4nVt2gwg5e3ORdql8K6xsJlBZIxph-NuPt1Ks2iB0ZarzoFELPAuAerVMiQpEU3nkq1JiqLAju26hE6Y2c3MwmKRTj66O8tsfQ-WFSh4d&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,P0xsXGcezpL-KzlLSOt_rfr6b6RF51B2hthJzXMGD04QeyLyg6AnMvF4M8ZOEwF90ojRdT3CtOy0x9HBSpGK_mmF_6Zl7Muz2TKrZRLAxQRxhV5XnvYlwQ,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,qctWvKkM1LAyegiROEl3hsiEN3dPcBcH4jlkBConOP-G2cgGl6X6XG-kdeQuBBOC8BmtTf-VmlbpJTjC_wznn8I6g7pLOg1uof2v0j_r3yyK7U8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,KRtNFryEe6jhqdT3of2FZC9PF7zGlYTxJ3KPFwO6TViZkyaHiybwHP3JmOhQ4F7yZtorUVRe5phyCswDWLc8lIFBVowIJygjwxLmO-F3xJSonSQ,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,ltH-LxKKSiKev4cs4CgS53B1Sn1z2t9IkxY_vjCKeoXMalJfF0yTujWPogG1a9qkWV5DrAT_qCTk7MdQa3LpkAae5yHTe_19fREF16p6vMMJqGXqLjw,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,66enqLb5-TVz_cB0ZCkDW4SpX4c3v0jslINMsxhHSjXfBWrIFoM8PxfSS7_VWmc_MdfmjPHIKcLm4zfcCe4MggwEWLxB-RjiUMqnmc8ShDJLoit5&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,xM_aKOMjg39ORE7gTvaGKP72ytDNvt6tjanfNTzekduYVGBvn963oijBiSY1JOQeorfgcCf57lVZSYIQ20x8sOFAH5zNexuTI0rBtrkDrtcJpq-S&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,Jr0OyBaxbJX-r7nzGvHtUkNxDkYucNHeuCPW_Mqw5I4IS0dve_XCjayhv7uvFQTdinHTsHqf2ZtZAn_qz1ZnE04PEsE0KWGfHQUj6N5WIKcNhyQ,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,fTod1kMxyNrLe6wmpPVQB1vco-7IrNIs9Dx8h9kWYDCH7xYH8V5kj2iao_X4D62DX5S24Xi08LA-xy2zHwj-LIzSKOA38iBBAlHdMD7qu3lK&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,37VGZcLM1KBT3tttPbLrhK822id0s0HCTimMYb18Jk-8bCbWxvPl2N9_yFFLgXa5pC9DXbKX6c8DMHolUFme8KRWVQdBx_lbRwtpLIOp6O7uiuV0fttkzvI3jxdO&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,zIgfFXrrGxOB_lKTtscVc_jXQlq_pH7TMnaK0IkixM4MVqoSgc-Wp5VV-QGQWmUtW5dggcgtYpRa7OoRHCBvDew7fxYJq28eSZVguCiuBiH9DzdKTHAXC5o,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,b6YutPqRZaL3jtcBxfqj_SiU4M_x_Qx-GeSmBfL7c6CG24IUL7AdjJxpyosXZtCtKco4bSDRxOj7HJn5CWgbgbPzJ0TFuXzJjs7VRUKVhg,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,xdw7r1BqGJuJPUUyo2a9wj_y2KKTC-j8c0DAocpHKzpZZaVOgPcxdng0Xi_JMKvlGYmcUvlTfZMm2FYQ7pTrF20xWUreNaRQqtU6rlWB3kkuPmMNBALyTBs,&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,4iCAaiDF74c7cCKTdUIIlT6Qu6BmbQItK-gaAIrQtpNdie21ccXNS_dCydALepUtQNTgsVsVUs1NavHOugg0IlFK0CjTj8TKnThm55ffUeOx8zHc4oZUYpE_LRsz&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,qBGyQwtcrkAjVKJZ9cZzaW1ivhGQgGfhpcJUS_ww-vfcQVNfD9OjDwDX1XjpzJWYpo08MqSOrKLpgO8d72TBVtSGg2OQ4AmggCv3IUgKeA,,&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,Mm6fpsoFsFLUtanrkJ6cqlk2y1zCJbPDIfqxsd0pS_sfS05oTAHBNIXIUUd4iA8HJY0guikOejZmykugHqKeopxLvJE3UmpX9YX1Y3Gu2ToK-B8aRw,,&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:34.435 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 54, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.465 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 98, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.474 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 20, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.483 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 128, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.495 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 158, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.499 [Thread-4]
> AbstractProducerCreditsImpl.checkCredits:184 [] DEBUG - not asking for -1
> credits on activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,BYjREAStKM-0e-e5OOY86CxL1g9WrAP01m5ROYtsGCMHTh3wox28vU8_r7Y9e7QwRiDtJ065Xst_zM3u0oFx_K7Fr-ZkN1GBxPR6aWEKrm4nQA,,&typo=1&ancr_add=1>
>
> 2024-01-10T19:52:34.499 [Thread-4]
> AbstractProducerCreditsImpl.afterAcquired:98 [] DEBUG - AfterAcquire 333
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,r4o5lwz6oA1VbUpicHhTmEpC9U7BwWBMw8ZYMcrl4g-jVFykL22Cb2WJIZLOqv-hLbxRqgThYlmO40Jo84JsPHe-IYBjUU8WMQ-jy1lZqxq3YQk,&typo=1&ancr_add=1>,
> pendingCredits=65203
>
> 2024-01-10T19:52:34.506 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 106, performing as a failover operation now and recreating
> objects
>
>
>
> 2024-01-10T19:52:34.680 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 155, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.683 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 21, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.684 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 129, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.686 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 57, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.687 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 174, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.688 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 176, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.689 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 56, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.690 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 108, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.691 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 156, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 233, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 112, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.693 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 187, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.694 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 110, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.695 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 144, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.696 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 235, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.697 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 28, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 47, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 30, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.699 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 77, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.701 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 65, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.710 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 70, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 34, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 159, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.713 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 76, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.714 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 119, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.716 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 59, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.719 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 87, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.720 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 80, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 166, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 149, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.722 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 137, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.723 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 163, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.724 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 179, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.726 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 181, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 17, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 58, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.728 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 32, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 49, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 234, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.730 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 143, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.731 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 42, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 136, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 178, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.733 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 55, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.734 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 95, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.737 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 48, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 16, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 146, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.739 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 22, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.742 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 43, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.743 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 130, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.744 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 36, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.745 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 60, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.746 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 117, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.751 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 14, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.752 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 81, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.754 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 93, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 138, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 100, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.756 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 62, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.757 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 125, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.758 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 15, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 99, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 44, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.760 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 172, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.761 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 84, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 79, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 122, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.764 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 64, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.765 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 88, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.767 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 151, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.772 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.773 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 96, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.774 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 27, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.775 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 73, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.776 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 90, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.777 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 134, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.778 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 35, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.780 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 118, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 186, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 41, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.782 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 31, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 150, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 145, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.787 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 127, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.789 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 132, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 170, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 142, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.791 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 53, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.792 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 86, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.793 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 147, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 115, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 12, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.795 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 135, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.796 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 68, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.797 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 103, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.798 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 141, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 165, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 18, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.800 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 33, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.801 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 116, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.802 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 124, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 26, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 50, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.804 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 75, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 85, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 69, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 97, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 101, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.808 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 153, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.809 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 107, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.810 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 175, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.811 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 66, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.812 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 111, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 152, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 45, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.814 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 61, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.815 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 113, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.816 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 105, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 162, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 171, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.818 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 173, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 184, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.820 [Thread-15 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.afterAcquired:98 [aoy5vnfnjetz,6ek4wlg8c0ad]
> DEBUG - AfterAcquire 781 credits on address rpdm.v1.authentication,
> pendingCredits=63974
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.user.group.query
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.user.group.query, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.user.group.query
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.canvas.state.query
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.canvas.state.query, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.canvas.state.query
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:184 [z7wx0g93sdr7] DEBUG - not
> asking for -1 credits on acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.afterAcquired:98 [z7wx0g93sdr7] DEBUG -
> AfterAcquire 505 credits on address acf62840-864a-4eb4-a902-4861db94eb93,
> pendingCredits=65031
>
> 2024-01-10T19:52:34.824 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d43c72b6, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> ClientSessionFactoryImpl.java:410
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,dWlUyTFJQxRIqOG8poa5pxge5mYVJ48dUXfACv6LUtKsZzFSVGKyZagOScZPGCnB6cxsyzdobSyk8B6Bwd79504jcLncGDwqMeY-9K6dpOs,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> NettyConnector.java:835
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ZNE9b4nkKfo7M-_v-eXIZ86i5mBDuVpXhcuhH7d4Z3TXAYIWBA7iz2F7WJKgkGmWz9vMjcTKd3WNKgAqq_OrNuwfMP9rqwnZZdfDvOywCznI2b71&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:666
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,ud4sg1bGnGSsxR72jGNWKmRRRqC9RhLjc5Fa_mAH3joB4qQCFRRYETYRnfp_427BSgD3-id4mG56z7G-3qi9hl8xOglhqXamz4f_kpydTW4Drw,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,5ea91cHeAd9MIKFvy9sbzhVk4n_LaL3aA10YRDmCls04v12YLb2XMmXw6eBUcLVuIE_Z644AmEt_e0ljvQnrS_I6HavZOLh4zjfiHWUvesg,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,jDru8ObAZFnt6UJuRwWa5fbG9hi1-FGT2jjgjYcb0j9MnKP4NE8WTbKBHSHyKvKnv60VBWqMlES8saZuop8zduWr_U-bg4uKdFl24xth0Wc08-CvUBpd&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,FCoOt1eEzJLmEhVMnIIHAKa-kTFceMfQD6Kcwc4ZSxQI7u6ROHuAf16piW1PXfz5fwuSX1g87RD5aUJh-U4htAPPBkPWT1OqrJG8TvqS_LoMCTdPQPVU&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,RozNWdaPDnLkxzHk0PIIu7yOI0mq7J36SJtiYqHJSj83AaF6otFCOMT6S0SrKP0i5_HdKBbJFOfyhgDON-XzQcdSOvC2Z4N90Agiv9ZXimQSY87tDA,,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3xUOFzb6MCi6u1j2LRQL6CE5BWbu5eM6Ir7EsfF2oCaYUBtqnA1r3VDYEmgBiVBb7QypDxbOzY9knH1MLZZ3oikFpY_p69_553LHOTbV7g,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,9xoBaA3kGi2K8zQ8BuNLDq0dYmsnE1nuJw9sP_cDKKpUiUl1eHBvRBJF127g2yN9LNqLqZkyPVSq8YusAWP6k6MCI4_4n846oL1LY84Tow2q00oOukNL4Q,,&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,OqfGI8b5WUrx4fzs6Vhc7lLcb55MI-F2H4j9ic-9Fbs1_FhBfTLzdobmjbVxUHuNV9kcphYOQdlHoTf8bN7hu4yIeKsdhwLW0rasAMuEm4mWhNV8t2c,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,ml--e7XB5_TgtwWFh6lw23pXHYNiKpjFGKoUKmwrOwHyS3lnFQ78O6WUjq-OM1ITxXTrgD6P5K2E89TnqOgGcaOyn6aLaVIcvPhARMyQEqlFDI2mcnYwIfwADKB-&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,olHwwJjC2TUJmGxt3e1EpgRMfVMAAdmudKrqZ5ZByilJc51VAK_ZZHGL1mLqUqv59mtQqrIVOT0FHxcuVGnfmTsHrtP_Lfh6vEjgZmLr0Aa8EqAKSU0BDQ,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,3crdMkhGhlkrJXp-HsjZIEJOjIoajfvvI_g-uTOHgEc26-rm5spEwaDR4tTrIOKeN5HSQz3HfKcnanjUMfuquaMfvk7M7K2AtlUVxZdR9Q,,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,Qv7RvLuSjcsV2_72qHHLM6lmOI05RppEGGyR8280SVuKWQgOAl-EeFo8ZDeB_9Hw2f5ZgHbKDJbuB11rT-FBX6wf71ASJsIGNZYXzHn1HnNPy3pXaBaEgvrJvA,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,7hPEKeOrrW0p5qUGMytIu7avFNFhxNhXFzKgo6a-hDqm4ky01dphgxAxTpF5DmP3oUrVV1uWYwMbJuUFv1IoX_o9-H7Y6xXWdj067c1A&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,dU7lk32qJcXeW-cUtM1E65BB47JiLhSgJYTqhLIX373Pj0miueYi3kN7MYtAU5zT39qOtzzAiHcBVYWsGDB3N8R31seI_xbm-rQzfd0jzb6V1rSLy1kotiI,&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,BtR0iFYtXaxmAlygBDvC74BLAVELzJivsFTs_YS41zS_XagPPAr0UUKW1lNrafWE3WxaZcW_S5WbNn583_uHQ7aLAA9vskUJfxaMwUrM_DxAOd2N&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,nW_o_ayhGlEOytZThvhNsFjqKDq7oEaYAEjw_jVKEHAnXglYBlSGSA3yj7wyG-2DXRXiRZSD11UwsKx7fSGPuHkUUSoULIYqacVjSrfmUuEfG_A,&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:34.824 [Thread-19] JmsStaticConnectionPool.onException:78
> [] ERROR - Receive error occurred. javax.jms.JMSException:
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(
> ActiveMQConnection.java:714
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a714&c=E,1,3FhMyZ8H2FihvFzilmw5MqmH_tkqp5vHbCOJsN5x6sQNSLyFELuJt_Ej2M_24CH42Xx0iQesALGDJK8HHdR_PDaLploHa8R84-uwfftA3DUM&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(
> ActiveMQConnection.java:735
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a735&c=E,1,JvpZWdXDc673FDsIBWz8WgGu29y6j8H0zAczvf_UXYyUAzEMKtvXJ3tMkleaFsrHU-QdlAUi0bWddjnikuurWEEMfZs0jIDWZ3jnrx2__Vbg93yZ8oo,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(
> ClientSessionFactoryImpl.java:868
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a868&c=E,1,AznnL1cSXDTbCj6iQ4ovEeP71-oopbCcH0laxV9dB-Y_kBuerNfeOAXRkRLybAqgqQBjJZKBNFEoveCJjTGncMCguHnY0hvsdZuKIHz6BEhvRX7N&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(
> ClientSessionFactoryImpl.java:856
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a856&c=E,1,vBewrTh4rfp_ezwl572jqeiy0VBShHwQJNRT99yv4B54VM5Qaf2hTHrOq33emXwk_WSFrcueJSCKI90AxcN4jUznfP1ZLzL1dJdUz01zK4vN&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:802
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a802&c=E,1,EY-ry2DzdJloXWtaemW-dWnRy1-rr2mQFFjoCbmWjX6xHCNZF95ldY2zUgfhOhSQFfzm7eYV0PdRUTOwJrx9jCD4U-B5ztFqWmvqEmJQuqdCyQgypA,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,gHouFFMo7plejV1zi4ureD9QEDkW-WjvytRlH6DOyEt7oMO8ULkj6APov-gDhxtXAspTlk7qF6RYw-RrCqfSrXlcE0Cm45J61qJHt5Yc3zuBJJyEX8A,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,17MGSyeFWTwZKhb2DCqWeyFvdlinlKAunc4JSHJltjjiaCeT_1UjXFsTMtEiF_NYIOMI3brJbPkRW1QkJIhe_Br44jy1od8-BI3l37IfFJzZbcculrc,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,F6NyfiAiJI9fxHnOQnnpO7dLyoFavqwnNivvLL6x2LO54G8SpNhXjlUN748SkNWwe_oCD9JhI7u_BOk1uNbwSTYV0lXRyewBrwvvysSEOSCggiFCFRgjtA,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,TnFdrKl6nCZv9PR6lSsRZ2YN0r8LqpGQMRo3oHlgyE7_AgUMzrW12R4HeB5JhX8Akv-1iKeVGrnTLV2WeAbBOCM-fukhfRPsdUgq5pwe0yU,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,vAtPtyOuzaufOELKjMWnOBMrJEOShH0EzH9G3WL8gak4hWHmnVWPL1W7c_YnxRH2IerBQErg9VJApOiiX6TR6tcPau3xqRHrPyrsH70KHg2v&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,pak9EX0xrMkRTN71voUljZIOmUvksqdCCqWdwh8fq2R4OZXUQFIHNH3P9j0t-wpQStfSs_ZsT0a9BZYopaoyMSnMkN3xqvXn8A9dTehv&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,acJEflBFObRkFNBskLYk7AARfS1_CzZH3yFGYB9r_Ype7n0DAWduZhGpmImVc9k1pTnmFoA8L8UukMVvWBinxujU6r75ipmzh4I_cNQ5Gzmd97g5SAU4dRsf3Hw,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,aM5tVNMvOtLUxuoV8Dez1Y7fAS8KwME3peX3IqP_QPViIWgkZQahJW_1BMBQUo35u5h9hGJ3XSyDgkECwWOGDn6a9W-gctjh8WvUKx9yQ0sSLwheMcs5mhR2yVM,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,fUWcgLX3hupjGCjPYEByZ8xfzFx4iN-VnkLB9MarNu6OEOfrwnCkvJvtu6e-ZV-FREp0Ssisi_fezDu18eP9adS4mmcPPlJH69vnf0YDVg,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,KuAjbB9H9ddyi6unfXrVdNXylpB7l6qLIUfJARScyUnc7iH1GtvO0s1MAtB9fwkdqXq8idOJYZs_rg0iU-sN4aV24ID0Os5tsyvB34ba3ve2D3diROqixzkQuw4,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,EcYd8WT9GtMN9IXo58gsQpj5uNwcdCRtDfa0zG9t2jpvfdaSX5r9vLey2icJ-4wo3EyBMyUQzwyqYwC5BMfe4yHGjyo7h8jxmjcEKig4V0KyVa_SVq58NBT8&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,_a7OW7NorEoCm9eOnYjCi6TFl2__or_ickK-DwP4rrMmRX2Rj_TKaxpGjSAwCJZasllt1gefhShoob68MthMyp6bxv9N5fucG2HiF0g3yK4kqHTZjiFn7kxZ&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,-ttWHQzQDZ94E-HqKtYTKV2nuApkTk60Zy25iWFIykqfNe0o2seEul4UE_gETZr08B40bJjmZKMHq_-R0M8bQ8Fbu8twSkWx9MPLFUhc&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,624T4PuCyLmsS-UG0IoXRx3MMz4DqnU_1BuWfOz973WSpqfLsFFDAICu75KCoCwAs9O8WgKTEcBVqRhhf1JUqI3Znxem3LsG7Ml0Td3DnJJUoGYR0BE,&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7UL50c_ahB0IhSHbe9OlGABBKKE4OFyNM67SjmNrSo_LQhfViHAlLgvwdlRpbmDdymI5VaeOEklHjvlweLBz8Ske6wjc8J3CzM05VYenh6DzNiI,&typo=1&ancr_add=1>
> )
>
> Caused by: ActiveMQDisconnectedException[errorType=DISCONNECTED
> message=AMQ219015: The connection was disconnected because of server
> shutdown]
>
> ... 7 more
>
>
>
> 2024-01-10T19:52:34.828 [Thread-17 (ActiveMQ-client-global-threads)]
> ClientSessionImpl.acknowledge:1225 [] DEBUG - client ack messageID =
> 10054445
>
>
>
> 2024-01-10T19:52:34.862 [Thread-24 (ActiveMQ-client-global-threads)]
> JmsServerBase.logRequest:117 [] INFO - REQUEST IN:
>
>
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,01YVdf0IHIl4w2xIwUh5i7ToRiQfv6TxT-FRWSNW0n6e2aucHkWVyKxkYJTpG1jt7_EUCuQVS0NWGalDyf11AUIb3qVPj67IUtjJB484IdLKrHgbmSd-Q8Vg7AKZ&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Tuesday, January 9, 2024 1:57 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Both the "replicated-failback-static" and "replicated-failback" examples
> run without issue out of the box on my machine so I'm not sure what's going
> on with that. One thing that's odd is that out-of-the-box the line 122 [1]
> doesn't match the method call in your NPE. Perhaps you modified the example?
>
>
>
> Aside from that there were a handful of questions in my previous message
> you didn't address. I think the questions about thread dumps are the most
> pressing given that you indicated you're deleting the pod. My guess is that
> the client is getting hung in a network call that doesn't want to return
> due to the pod disappearing. A series of thread dumps during the minute
> delay would confirm or deny that.
>
>
>
>
>
> Justin
>
>
>
> [1]
> https://github.com/apache/activemq-artemis/blob/2.31.2/examples/features/ha/replicated-failback-static/src/main/java/org/apache/activemq/artemis/jms/example/ReplicatedFailbackStaticExample.java#L122
>
>
>
> On Tue, Jan 9, 2024 at 12:47 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hey Justing, thanks for the help.
>
>
>
> We are using version 2.31.2, which is pretty recent, if not the latest.
>
>
>
> With regards to the log traces, other AMQ threads show similar behavior,
> and our system isn’t doing much until AMQ-client stabilizes.
>
> I am running on a k3d cluster and the failover is triggered by me deleting
> the dm-activemq-live pod, which k8s then spins back up.
>
> Also, our slave config does NOT specify “allow-failback=true”, so the spin
> up of the new dm-activemq-live instance does not trigger a fail-back, which
> is intentional to eliminate failover-failback chaos in such a short time.
>
>
>
> I am trying to setup a simpler test case by using one of the examples,
> artemis/2.31.2/examples/features/ha/replicated-failback-static, however, I
> am getting this NPE on line 112:
>
> Caused by: java.lang.NullPointerException: Cannot invoke
> "javax.jms.TextMessage.getText()" because "message0" is null
>
>
>
> The NPE seems to indicate that, you cannot call consumer.receive() again
> even if NOT acknowledged, which seems to contradict the comments on the
> code. I have run into this same behavior in another example as well,
> artemis/2.31.2/examples/features/ha/replicated-failback.
>
>
>
> Can you please confirm the expected behavior of the example code in the
> 2.31.2 version? I can then modify the code accordingly. Or please point me
> to a working sample code.
>
>
>
> Thanks,
>
> Lino
>
>
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,lIOPl76F2R4EtenIAEJMeQ17kfmHQ6buC7G1W5TRefOcSfxvj_-3dLXV3MKDkCTkac_65Rw0SYmof66i91zGwgb_SS1nKaBpao_Lw7JNNk1mkE33&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Monday, January 8, 2024 9:20 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > Now, the backup process does start processing msgs right away...
>
>
>
> What evidence do you have of that?
>
>
>
> > ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute
> apart, before it stabilizes after the failover and is once again usable.
>
>
>
> The log snippets you provided show a reconnection attempt back to
> dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to
> dm-activemq-backup-svc:61617 both of which fail almost immediately. Then
> right after that another connection attempt is made to
> dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e.
> from 20:03:44.874 to 20:04:44.933). Are there any other logs during this
> minute? Did you happen to get any thread dumps during this time? If so,
> what did they show? If not, could you? What was happening with
> dm-activemq-live-svc during this time? Perhaps there was something weird
> going on with the network related to whatever you were doing with that
> machine to trigger failover.
>
>
>
> You might consider simplifying your test-case and then adding complexity
> bit by bit to see what triggers the difference in behavior. For example,
> run your test with the client and both brokers on the *same machine*. If
> that works then move just the client to a remote machine. If that works
> then test the client just reconnecting to the primary broker upon restart
> (i.e. no failover). Then finally put all three entities on separate
> machines and test failover. Hopefully such an exercise will give you a
> better understanding of where the problem might lie.
>
>
>
> Lastly, please indicate what version of ActiveMQ Artemis you're using.
>
>
>
>
>
> Justin
>
>
>
> On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hi Justin,
>
> I’m working with John Lilley on getting an ActiveMQ HA configuration
> operational. Our current HA config is replication with master and slave
> instances and with static connections, and it seems to be working, except
> that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect
> itself after a failover from the live to the backup instance. Now, the
> backup process does start processing msgs right away, but the
> ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart,
> before it stabilizes after the failover and is once again usable. I have
> included the relevant log traces from the Thread-0 ActiveMQ-client showing
> the Connection failure detection all the way through when the client is
> usable again, over 1 minute later. I am hoping that, based on the trace,
> you can find something that, perhaps, we should be doing differently in the
> client side?
>
>
>
> Here is our Client connection url:
>
> (tcp://dm-activemq-live-svc:
> 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
>
>
>
> Should we also be including the corresponding connector names?
>
>
>
> In an earlier email with John Lilley you recommended 2 brokers using
> “shared storage” instead of “replication”, which I will try next, however,
> since this issue seems to be on the client-side, I want to make sure our
> client-side is setup correctly.
>
>
>
> Thanks in advance for your help.
>
> Lino
>
>
>
> Thread-0 log traces:
>
> 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-live-svc/10.43.197.248:61616 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=a8a4a77c, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> ---
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and currentConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.197.248:61616
>
>
>
> 2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.11.76:61617
>
>
>
> 2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> ---
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and currentConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.197.248:61616
>
> 2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = ced14fbc
>
>
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.11.76:61617
>
>
>
> 2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
>
>
> 2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 10, performing as a failover operation now and recreating
> objects
>
>
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address
> activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,hqiYs6vnJzfq-X42G9W9ohJtYgn9LRPR-82RHxMyPnKUD5xBwCbo8W6b-PegfmAGwvEENlDO0k2IkoGFFegGxR-H6J72UHHlpEFbGtJzhCF3sucQ&typo=1&ancr_add=1>
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,KX9atZvl9XxpGPWobW9hXsmaLrVekfw2221zvD0PJ9Ik3fEQVyHKcZZCe7MQRciamxD-T-WDEk5Xa7_cAt0PSHwVW3jG7zzhq8XH9XIRC8EXmg,,&typo=1&ancr_add=1>,
> needed = 65536, arriving = 0
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,ZOCwWoQv34cYLuw9hZvJzBHfSrX6oQeQ_3-SetFMJa1hRC6lXgiKsAElOhfZ_vfnBhRGRJPZjSOqHIbwWO2ltSYYY7ElQjyItp0uDo79_jaDkw,,&typo=1&ancr_add=1>
>
> 2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=a8a4a77c, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> ClientSessionFactoryImpl.java:410
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,obbNTaFqn6mglgyZXy1S-0nBLNVtEYlkZBwAEKSGDGN0RuqlR_-7fIQPW4ELNqVFJdenAcKh65C3yQ2b_Lef7W01Tst6ZTrmI66KfHLae572jXv-HF3O&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> NettyConnector.java:835
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ytVVTXzfq-J1NXl2ct7T_47F5EbfZNQTXP8dlqw720EiCe32c6CLwlpk0-v4MzUD61FErq0nkRTalFpcTtIN8TDeBvz70rzb__TyeIDm48ZKfFU28qQ,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:666
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,KQYUuEgOaR6V8a_ZC_xBpKZmDneB4LTL1cQzn2Gqtla8ntHJ1fgRs9KelLgvbMd9zvBTg8maXUerYZYQZa-DlfK7fon0JZ0O74PNJ1SOCHqIPN4fZQ6Xv731OGOD&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,OSVh6NSORxz7qZ6eIuUDgWskoYfyXT4NNdjJYGMbH7HeWEUiymys2DnptjYKdmQfqjfHmI-egRZ31yWwd9MKzEd-gbbsuTUsqn_46niKaw,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,DM2boEBGg44zzcd0m37ITkOnfDrkqLE8ZSFGJpEI4AGo2GyOI2yKX08AEkKHxw_UnDMF1vccYEvESFDXjXBBjjIGO5TEsMLWZHbJF3AOIEQ73uc,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,wipgGeA7UVZIy1uuqvuxJEx4BA8TvYfSYmO617IgrvnOo-z7sn1vUjXFqIEU9M1HmedBVoHxB_LpvuEZ9p_biioPM8V-V9eS8i8oST46GvBhZZIc&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,0FNTjscT_d2CNIid0J8d_x7C7Peb6_neE-a_MZeKqkM1o_sPGujvpkXuiLv6kms9wSFyYqSux86C1MX2R8WKM8IZBsjKt7sCs03U2lRYlVbw2eTDg0ihSEV1nA,,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3yKPqxpIi9n43QJSCQiysKGlEYttZ5Dmuirc7jUAWNz6orLfIUCYwdDEJ3eU0QiHywsNAJdWqrvnx5rZK6-8TTJb6o8tr1WSvSL7omLF&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,6B4qLrj7q9DzDb0Trp9xgst7cOBhbD4hcEcGyPFul5E_mh3_oGmColB9YQkQTB1gJ6x5Mcm0VwvBLddpDpfFkcZwlzb4YNR9Ym5fcDclFW97pIrQ9oG4&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KqR93A_gEw1FpU7__PsYjSkOC5LH0zFTSKu5K9YlCdVAz3h_c9EkBndO7aliE4RSbK_qslEZp0b-YzJfCOxmB22jAfplsYxguMl1qjC0rWpNjfr9_fh4RjNL3AOK&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,mhChlKkihYjDw_Kk5SL1d_4eb7zhBlSzri_nwjOUsrs6MAvEfDAAjia07yTMUU6CQqc0WbDS2Fn8HJJI0UwJh1N7SKiz9dsP1gJLmgY66huLhXZE9c-3ECGXyIWQ&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,nn8IDmg6XQJLLXzH7kunXDNZa_3A7fUlnHdRCLoha7gQWJO1QeY32duGFAQMYaFHsfIbHbv142mei8HfCqRwrpuUfikYxLkFoXaD3pORZb9roBJs0H8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,DPLEvt-MPmmzv78LEkeXtYHM2q37m12TxX-nqjuIJurQdeXydsSakco9t_O7jgBbQ6y115xxdDb8IonFZbM86zgRDLcvUmeOSJuDmZybD_6JaTX0zpbQ&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,1vNFV-mDaLK92lJo0Mq6jvkCDzuwrSHT4lp1nB6KTzOCSO6comH5xNpEcvtZ_9pc0cQCD2kgDfB5xdxg0tAX2dF5oG5DHY6kxWXOBLnuAwiz1BkEFhw,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,aLydz0SGsclotqCro2upcTg44B_Ij7ckjApLu4B5JUwZKKdMo5z3RyUR4ikHFY8L3t_yB87Zp5atGRwoAPA1L9Mc9c681UjFrG16n-udVjeHtxQNxx8V-Nug&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1mZX-u-XHEZ_e4Ut7YETRvWm7U_qkC-q9QaPeHah4K4yNl5foyM1VSExCtYiisknkelDj6_K2MUHGc65Kd-6-JG_TXLby0OPWodJVjX3BtKaDNBUzusgpzT6ajL7&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-nbEPDxI7J-DFvbJfVVt2Cs1Vi_hN5bw-8vQF3SlM1nthw3VbJLWmS1h73cupUxqBoHCImTzDqI1m5gutx3ZriLpLpU7IqutHCqZuT6CTVWACPtj&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,-lUcdhDzaHWdAWLCc0IxWCFkKtKHxoEd64GWhzXAbeuG_Zm9mvn624BY6uS8MgQn-s2TMF8PbEPYVbOOy6BJuI8ByeiuAF9Y8f2I5wib&typo=1&ancr_add=1>
> )
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,T2UV-1dji1RODRcU4Ff4j_IzfTH2n47BatzyLZcEOU8VROFr3GS8re7CjDcnnHCSaPEtR-35J3z30dnYcz5K66rVsMVxgDiyh0FvEDKnN67naHjYbn0NpzAc&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
RE: Looking for HA/replication boilerplate broker.xml and advice (continued)
Posted by Lino Pereira <li...@redpointglobal.com.INVALID>.
Justin, good news, be it somewhat inexplicable.
I had to apply some updates to my Linux VM, which involved a reboot, and now, running our system on my k3d cluster, just like before, and failovers now work in both directions, instantly, and with no chatty exceptions from the AMQ-clients.
I’ve done a bunch of failovers in both directions now, and they go completely unnoticed by the MAQ-client, smooth as silk.
I am guessing my issues were due to something environmental in my system that got resolved either via the reboot, the update, or both. Perhaps you have a more scientific explanation?
Anyways, I am both very happy that this saga is over, and a bit annoyed that I spent so much time on this, but, in retrospect, I did learn a lot about AMQ HA, so all is good.
Thanks for all your help,
Lino
[rg] <https://www.redpointglobal.com/>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Lino Pereira <li...@redpointglobal.com.INVALID>
Sent: Thursday, January 11, 2024 4:11 PM
To: users@activemq.apache.org
Subject: RE: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hey Justin,
So I am trying to verify our AMQ docker images in the local environment, basically our system runs as simple processes on my local Linux system, but the AMQ instances run in docker containers. Now, this works fine for standalone, and for the HA case, the “master” instance works fine with, or without, the “backup” instance running. However, the “backup” instance doesn’t seem to ever start up fully, that is, it is missing the AMQ221080, AMQ221003, AMQ221007 and AMQ221001 log lines, shown below from the “master” instance. As a result, it never announces itself in “Backup” mode as seen when run from k8s. I am using the same broker configs that run fine when AMQ instances are run natively (no docker).
Also, when I stop the “master” instance, the AMQ-client reports these exceptions and never connects. Any ideas on why this may be happening for the backup, which uses the same exact docker image, just with a different broker.xml?
Is there some fundamental thing that I am missing when running 2 AMQ instances from docker containers?
My goal is to validate that our AMQ docker images are functional outside of k8s.
With regards, to getting you the thread dumps, next on the chopping block, can I just attach them to the email, or is there an alternate path to get those to you? Apologize for the dumb questions, but I am new at this.
Thanks.
Lino
AMQ-client exceptions after killing the “master” instance:
2024-01-11T15:47:15.322 [Thread-0 (ActiveMQ-client-global-threads)] client.errorCreatingNettyConnection:741 [] ERROR - AMQ214016: Failed to create netty connection java.lang.IllegalStateException: No ActiveMQChannelHandler has been found while connecting to localhost/127.0.0.1:61617 from Channel with id = 0bec947c
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:980<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a980&c=E,1,9sVwvAhg-dCUoWlCTkknJyr35Kb1lORNDVxR6-uqw3mazBhZ5pG19VN92yx8WmUo86gmEfiM-fyes3vKIRRqmUr7xGxTepKfWp8f8YCxIKDeU3MjhJIbTwA,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:866<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a866&c=E,1,PDO7x8ScbX2lkRczVyc7Qx5qFYh8jYI_QZSxIyD15vNJbt-HUSyFIAsL54Aas_i1kj247doxU1D-TF1f6UNtgifP4lOW0jVh-Ca8bmu8X2BRDUZyZYrlZd4,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:848<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a848&c=E,1,M5S848Ei6ypG34y6nHCsxBxodJIiZs0WdncK5b_Paenx_yRNXYZToN-BG2GJ4M8V03LsOGQzIKSbFW7uKF8az4WwG4Xrzk0hIvgvf05YYBR-KraWq2Quc6cO&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(ClientSessionFactoryImpl.java:1212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1212&c=E,1,Zu7gFKlm4HmSOURefrcnPu7_omfp3z6NkhUulfWOlvCON5qFl8o_m__R5jhaoNXCjQUWg3afJOkgibeky8zBPzqugdJrnx3mDcRvktMSa8mHPzzyhu9E&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1333<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1333&c=E,1,QLuXIqCAvTMp9vzA3xlqCzu6uu_CAV6-6B3ZjST9QtAqqlHPPW031VktMnXrspcOD4IBb1JKvR2F8OzNZIi97M8NdBmUvk48uDHiFLvxVM4,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1253<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1253&c=E,1,HGg4Q1mxe7PWvtXuKXcO1vTG9pGtPTivn_BSZ-cDFd_RB5T6KlxMhp2Z2iKrHRFFrcmAJyClu3xTgdygcYT1cRvfTIU0mnOfoNn73hij&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(ClientSessionFactoryImpl.java:1496<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1496&c=E,1,YDsqwgPbq8QPr_ODsKRUE5-FJdCxEgI2EdJdlE_ogGX_Dtkdqu0ZUW5FnNQHKMwQfJEjscTg0sXGDaY7D27_1M-BNdURAJybIJ0mM63IpSP_Kix0E7dJj0xAAA,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1074<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1074&c=E,1,UaHGLwmlvYxEp5Zvya3OgHoGpFPLrtGO8817O18DQ_eCtU9hppVQz3ZSqMOZR4Ou0RyfDnJ25Dsho8qeOiDziqkIZ-8smasmw6AS-SC3C07T9A,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:959<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a959&c=E,1,S2vAOnIQrU8itB1DHvtgyBp5ozbblk23_CGr3q-ohZ6Nt35WRISFB1PbqeDDdjRyaj8N9tzZSNUm_TpeEwNQg143NiFeTG18rXYlnq4jakQe2_H0HkGTrg5Qow,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:697<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a697&c=E,1,HRBbQntC7regy9ruiCBdgTxHOjjHiqqEtdrKtyuFoRluNvypXTYG7MHAazXY4rr-7iE9ymvG-ccC_EuBMTBG5AjelMg7KfPvqiIaUkF1RfMIvoeiA8o,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,WwOwlRDdAEBeA1Gg0ArmZFo1it0EZxIt6RsmaWdNGx6wo_Lz4qKdMxIHy7U_9S5bgl-33cfGzNtlYG9YM9FZXucb7-SjGMsXWWRiyGWMdHRJ&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:559<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a559&c=E,1,Zc7KWsYoeLt5E_P6hgERE60yuujJHXiWpuT9jFjj4gC367dihQ6w7V22qGaDAek1JHPJHKEsYT6vqwQAPm9GSln6px2RaJkIttMBpwOF398rKvWxlzAOmI4,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionException(ClientSessionFactoryImpl.java:425<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a425&c=E,1,BFL6YN5Kxy06vezMnsiL2nC8cdKvlemlxhvuHQ-FuWkoOzxoYzQuft1CeyeiQmEo-L40YbRsfc6dzmYWbWpQQb_xVyaLDbql7YwiFlA60a1xj1PSDmU,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$2.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f2.run&c=E,1,sMyuWkYLp3LX6ibrSEXRcAezwbBM7qAOluPp_GC1jVesMs6jxDzxY4FFM3VjeigvvheAyp3Q7Oe5ACoqft2B-abCTUa6WF1bt7Uh4xjE6hHhWxo,&typo=1&ancr_add=1>(NettyConnector.java:1252<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a1252&c=E,1,3JJSPZ_KDZLXcsGld2w64FmGt132PiKafjM_G1ZXb-ulOdS6Dwv6XUv8QHJqHN6CKkClQ7lUkNnakzibZdks8vAdD3gLbxCOyjzJGO6K745CLD2v2F7UK5zJSQGG&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,bpAKAylKI6Drp540buFl0pkNNpNnX4xnMjY3zWwpNyF_392duOqGQPjpdfCPGxrpBvc7_eIyfJpxl4WuufWnNYLFKdIg0Nu9WDi6STLxtiY0tUhT7eeO2iA,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,CqAtDgvRoDEnuFjXXv-RSPQ-NcQ5-m4IRpyQ9oiN0rhM40sqLFbhY9BMbc0HKKA9gNrRlaFd_DPaSB11LNOv4Rdahsd5fJLOqXHsZnvg3ShSJw,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,C5wuD-LRtJ7605p6qERurgxxNLnHBKbC4B7VE9lfDl7f_JsOdDSkeiyMkYACWnhSckLFmjUW2eFKZawLMNYORn4gTtULbgzlBoD8LWUyNto,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,rMwMgnKWBZ0Z_t5VCFvBOA7J5McS_4eMRlTXeB6c0yMri_CRxIy8PDWJRjrRtRMI1QpqaQtuBEcG4sCoeTP06BI4NmwaneyL2iZ_EcqSA0sDvg,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,AmU9mOKxf3Tw98cD6iOKGEUsELr-mALW0vP5N2eC_TxCQIG4d3rxa4xf_W2vXyThIPw3Zmp6MJGpD2c95V3s3JefMOVZJGioR7UrvoKjJHS1Qr22gQ,,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,uKLItaPZw48N90kGuOJ2giezmgZl49ttEaMZ2H4_Lr1mqtycK7JXSo7FWi90G25OhrIF9k-OxzV8LaDSYfMSQPd4uEksOAIsnz2-SSjTNFctPDccNQ,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,gIn498Y42-KQL16Kw7akU8ywkXBbJPMfxg7-Ivrxphmo6NJdJqsbcSdPEAdkwVYS3FgjewkEVfwxLXUtcBsU4QHUbsEaVo5060hl3qEBmZeRmUBspRNTPB0,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,KRj2MYUCdG64pKABEvud93N21Qgh8ET58T7SDN4ZzK9ErKl92F_UyzJtq4y9eDIqY04euQg_cRGOTddiPcrePTpRRyy9U7AEk_WKzWUcU_VDwdQacLRCEHF4j6ag&typo=1&ancr_add=1>)
-------------
Log excerpt from AMQ docker “master” instance, showing lines missing from backup:
2024-01-11 22:38:54,844 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST]
2024-01-11 22:38:54,853 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2024-01-11 22:38:54,905 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2024-01-11 22:38:54,906 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols [CORE,MQTT,AMQP,HORNETQ,STOMP,OPENWIRE]
2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.31.2 [live, nodeID=320ff011-b0d2-11ee-a54c-0242ac110004]
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,WIQnILFOGT12V7T2V5JgdPTF8Y-yZZKUo4gSIgdk9Xb0RKhobbQh-PjrSOCrn9J79hRQkhxWG4u7HGnntjzO4n-TgjsgxAzGPARQB9OM7Zk,&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Wednesday, January 10, 2024 6:57 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> ...I’ll have to instrument the code for that, right? Or am I missing an easier way to generate those on-demand?
I'm not clear on why you'd need to instrument the code to get thread dumps. You can generate thread dumps through a number of different commands, e.g.:
- jstack
- jcmd
- kill -3
You can execute these administratively or just set up a script that generates them every so often.
Justin
On Wed, Jan 10, 2024 at 7:43 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hey Justin,
I did finally get a chance to run everything locally, and it all works smoothly with failovers in both directions and the AMQ-client recovers within 2-3 seconds and with a lot less log noise.
I did not get the thread dumps yet because I’ll have to instrument the code for that, right? Or am I missing an easier way to generate those on-demand? This is happening in a k8s pod, so not easy to connect with jconsole.
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,P1-JTENl1pkpwfayJeVxTcbmyxoXqv5aWc8dgoVoQa2P9YJ6ITYanSwPPWI3OOvd-1ZuyvCfDdUgZSHNKSzXPNMT51ZJGff0MJI1JNYM4KnicxOXm0JmVYcKXz0s&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Wednesday, January 10, 2024 3:02 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> So our issue seems to be only for failover from live-to-backup, consistently.
As far as I know the client doesn't make any distinction between these two cases so I'm guessing there's some kind of environmental difference that's making an impact.
> Is there something that we are perhaps not doing on the client?
I don't think so. The connection URL you provided previously looks good.
> I will extend one of the example codes to use threads and sone of the AMQ-client wrapper code in an effort to isolate this behavior on a simpler code base, as you recommented.
Before going down that road it might be better to inspect the ostensibly low-hanging fruit first. Did you happen to test everything locally? If so, what happened? Did you acquire any thread dumps during the delay as previously requested?
Justin
On Wed, Jan 10, 2024 at 3:04 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hey Justin,
After starting fresh with version 2.31.2, the examples work out of the box for me as well.
And, after ensuring that version 2.31.2 is running everywhere, client was still using 2.30.0, and refactoring to using the recommended shared-store HA, our AMQ-client is still behaving as before. That is, failover from live-to-backup, client recovery takes over 1 minute, and logs report what I posted earlier. However, failover from backup-to-live, client recovers within 5 seconds, which is the expected behavior, and our system is happy. Please see new log excerpts with more logging included for the 2 relevant threads.
So our issue seems to be only for failover from live-to-backup, consistently. Is there something that we are perhaps not doing on the client?
I will extend one of the example codes to use threads and sone of the AMQ-client wrapper code in an effort to isolate this behavior on a simpler code base, as you recommented.
In the meantime, I thought I’d ping you in case you can think of something obvious that we might have missed, especially since we just moved from 2.30.0.
Thanks again,
Lino
Our versions:
ARTEMIS_VERSION=2.31.2
JMX_VERSION=0.18.0
Logs from failover from backup-to-live:
2024-01-10T19:52:29.588 [Thread-24 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d7ff7885, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,O0LJ4VfRCemWEaMLULajHsgUo6YseAVDyBlrTm2G50vM20WhlTUV0ueY1DxMRAMStXAoc6wy5Xx3x1vi2XsnE5F7_iNbL-MZ03M4IS9G&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,VdcejKqF9B67eWPDv-PP_0gR6PEmUxFYHHtW8kyn3Gzg3q-HHzL-Lr7oztjj2LQumPJWO-1KF4OPavU8c3DW4zPgrUsidhnnCpoMUUn_&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,xKCDuAN84vJBjsCp75epCz7V1lR05TFhETdvpPcKntBTPnQk0zuuQSei7-ly1RPUUsHDP8ZAe-IevqwGtTGfEB_yNzw90QcOpEx5BaXOW6E,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,pq7BenEb7b9i-FYz1LnTy-gHYDdV-28FiSv3pkV_i-1TQZtRxakbT-rOKkgLgz2IDx875Dk5EE_OrwsUcew9TIsvrcZZZ9ZIlwboxMEq9xnR5odpQL8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,8mmrZ5alRiVlyEshqjlGpedpt7TuUVBJOy8DDm2qHLGoY6v3pR76llV59sMsYwy2v4rwLqiD9Rcf3Hg4QSCZo7rdO1rQgMNapCKji2jg7T4IJOk3OAeLnIs,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,RLTYzCoyknTJ1GB9szMq3Jg02QAnt7juedv1HCTYmmA4JU1DO6MlJhkvzOTBqHbeeOdKSvwkualBrhGt0fUk7Nl5YLzlgfepqTWcDiBmXQ-_xSMcAHPw&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,iYtEGEWoGf58Vq3CGAxBrH3iWZ1jWm1N1b3AOFPNlx0kCahcxHEwXT7ZAQ_h3ynXTeKJSkvmBiGUkbqE3bH7tGw-Gn8VV7R7LaCkxl9Dch5I4kQDfJDKSg,,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,kH54RADAl8RcHDeCxJW9b-P4YJJFElj_Nbh6mHtmwlFdP2tKzzbK3wkAXg9tVqhIlsbd4gr2LJkn7DWZ4I-J8WVE-7UovVbVakuQTTxc2pbR9vuIDiEFztc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,V9-XUYgB7GoqPyGhEdE_CmtNood-yvXw6P9wRIqQOSYSWuTfiF1bhonQejBQfgJdHq8k2TsE1RAvcSOp_WzBE-ybl0l6rc5pIuMGdC1LUITpdk_Iuha7urSje4Y,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,RNgnFH7dBkD27TX353Mc-kqneOLcqFXEgpGywxg8VduTTYNyYFt5VqkKfirozQw9V88WOI8XK4v9dvrtPIl-3u7cp673GvKxq3L-XgqypuDtSnkRIYnh813U&typo=1&ancr_add=1>)
2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d43c72b6, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,L97BTn6AFLbUpYgHzAlQm9pDJ4Sn--TlGDpz05xOE3xd-qn9krfLJ6JzGTaiSpQwuVjuQ8KUPfUSW6N18pgw44-ay7eqYb2h0p3sxehZeXAC&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,PiedcDlDv-nxD-EF13k9j-sagD4clTOVvCs6xyDQvnrPzDUxc4xAbnHg0d7hnqeG4cVJZaO42LJvoNm59E3SmClgLAft-3N7F898HO6Obbkm&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,G5SZ7tGYpGBQzNRxdmIzsp4hWlUsmbY3_881k0N5lFoFBwuHt1dMCkKHI8wiZRGpS6qDY_UDBvDmKVQNRIHXIIyg5Df2nnWDCTLnbvWCO20fW0PC&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,MsagygznR7ie9qgJLlMogXq8ex-X4ST-fg_vbVbgcBanXur80DVykS1ftnzq0e4bWLE0k8EiNmlhlezemqTlBEncWMJz-AQAKdTxEk4WyVY,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,n7l9CUpZZbFH3ca7Iu2wxM3kNfged80mMRcu0Vn9wIoZz7aKzCP3c1OVR73iqF0oLbir7CjMHKAu3sfrR2cLs-EP7orqDWQKQApkzTZra9CJag,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,PePvk640nUsvwJWNJY39J2R954rSgiFVPPSQLOsjYskMxtDnDVBuzQiZ8ut5SAPoqzV9uYW6GX3ekOAAoou9eu6VpHa_oFc2-7d27W00VBLU_o_Zv4wEUw,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,REmfZe55ytZRsqZvFkCNsYUcRO5Yd8PZwIlQfGystV0qUeF2mXSYaWKgleI098Ac40PND4ilF-ukqEWDY8N04AIRQgGgKGpG036xx7jNSc78YS6UbKWiMNUHfNc,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,nnqfN76WKNs0ZYhVPzea17QO5aWkwmLqbfYrmQJ_qGQIgTzCL4PSXhjU9jy4mhbCUfl53cwF4ISYabr_zD1GUvCgv5m4dYW1_OROCBpe&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,vlJ5vYZAfrlVp9kL50a_jwWhqVW8-LMUgzeBb6mn-udcS_fMR1yld889bXgOXTMDiutFqKYX9XPh0RtxeXRLtnN31WqeOlH_CZv4_ZyhnOwXvm4J&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7Ba-VAkMm3x0E0f6F1f73TH_y9Ylo03hx9DEz4kqWzvrXy0gAv3D9zIM35eXIaB95UFxo72DkMpb0nKW_a-BIF89JobbRqjyVNLdC5K3L1EM520B&typo=1&ancr_add=1>)
2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:29.592 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:29.594 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:29.593 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:29.597 [Thread-6 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d4cf19d5
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.600 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 7b2392ac
2024-01-10T19:52:29.602 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.602 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.603 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = c1fb4c82
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.610 [Thread-8 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 06b7b3a3
2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:29.611 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 65680be4
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.624 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 97a109b7
2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.625 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.629 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 77b2de2f
2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:30.612 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = f92d7cd6
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.613 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 1d44895d
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.614 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = e8602acd
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:30.631 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 0a217bfd
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.632 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d2aa00d7
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.633 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 9a4f639e
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.634 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = ecdb1cbf
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:31.000 [Schedule-Runner] SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service as system
2024-01-10T19:52:32.001 [Schedule-Runner] SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service as system
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 2/200
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:32.116 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = a6b3d488
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.117 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = c0627cd9
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.118 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 71523eb3
2024-01-10T19:52:32.119 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.120 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:32.134 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 2/200
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:32.135 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 68efd0dc
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.137 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = daddef73
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.137 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 84813ce3
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.138 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 533990e9
2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:34.370 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 3/200
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:34.371 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 1d8df71b
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:34.373 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = de20e193
2024-01-10T19:52:34.373 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 3/200
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:34.390 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:34.390 [Thread-10 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 4661fd5d
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:34.392 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d1c14a74
2024-01-10T19:52:34.392 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.404 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-10T19:52:34.405 [Thread-9 (ActiveMQ-client-netty-threads)] ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up backup config = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc for live = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.405 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 38, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.409 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-10T19:52:34.410 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 37, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.411 [Thread-24 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 10, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.411 [Thread-11 (ActiveMQ-client-netty-threads)] ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up backup config = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc for live = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.413 [Thread-24 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,S5999ok8Om6JjNfBLhC0R2hYdl3dGqPWsLpis0MTx38AhKXPNqMMlH41RrDz0UZsxZzSZbLvUEOYFEa7hwM4N7gAZfCS9udcqSmOwhLg&typo=1&ancr_add=1>
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,CF5ZPcR0aBH6uaRjLjSeWoBKLnRwUOOgvrD5MzWuerKGIs1FEa5vZF8rdF1kWSKbLwGQlm1IsOeyey0f1t7eiA1ApUgs1EDhL7ilSV0c6rFxEanucXI,&typo=1&ancr_add=1>, needed = 65536, arriving = 0
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,nl9VlpvL5Lx7cxWUON3eMxxXV1my7fPq5WQLaWD1ozszhEPr1wVfqCwFilScj02MzRACloIhLZ8CgyMWoa_OqCu-Tf4folOdvKvtooRq&typo=1&ancr_add=1>
2024-01-10T19:52:34.415 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d7ff7885, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,awSP7U_0_MIm7Km9lwe59SrlpKsd9iUMA2y266CCeNZS6GJuEExdDYVS6EVMnlmPlVN0W1aqSx2Rss2Dm_YIM71s5j7OAUz6pAdGaTqnU_h2RyaoD6KyuoU,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,8gAKfG0MZn0OMdhCHJ98evOezuGoCS0mNbGWplSwawxhF0ed3tPq9vKWoWUbn2xmoLJ34skkCTpI2tUuevSCeVNT00tNpFlOJlyAjGWu&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,tXm2hjddgUkJac0QpMg4nVt2gwg5e3ORdql8K6xsJlBZIxph-NuPt1Ks2iB0ZarzoFELPAuAerVMiQpEU3nkq1JiqLAju26hE6Y2c3MwmKRTj66O8tsfQ-WFSh4d&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,P0xsXGcezpL-KzlLSOt_rfr6b6RF51B2hthJzXMGD04QeyLyg6AnMvF4M8ZOEwF90ojRdT3CtOy0x9HBSpGK_mmF_6Zl7Muz2TKrZRLAxQRxhV5XnvYlwQ,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,qctWvKkM1LAyegiROEl3hsiEN3dPcBcH4jlkBConOP-G2cgGl6X6XG-kdeQuBBOC8BmtTf-VmlbpJTjC_wznn8I6g7pLOg1uof2v0j_r3yyK7U8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,KRtNFryEe6jhqdT3of2FZC9PF7zGlYTxJ3KPFwO6TViZkyaHiybwHP3JmOhQ4F7yZtorUVRe5phyCswDWLc8lIFBVowIJygjwxLmO-F3xJSonSQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,ltH-LxKKSiKev4cs4CgS53B1Sn1z2t9IkxY_vjCKeoXMalJfF0yTujWPogG1a9qkWV5DrAT_qCTk7MdQa3LpkAae5yHTe_19fREF16p6vMMJqGXqLjw,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,66enqLb5-TVz_cB0ZCkDW4SpX4c3v0jslINMsxhHSjXfBWrIFoM8PxfSS7_VWmc_MdfmjPHIKcLm4zfcCe4MggwEWLxB-RjiUMqnmc8ShDJLoit5&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,xM_aKOMjg39ORE7gTvaGKP72ytDNvt6tjanfNTzekduYVGBvn963oijBiSY1JOQeorfgcCf57lVZSYIQ20x8sOFAH5zNexuTI0rBtrkDrtcJpq-S&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,Jr0OyBaxbJX-r7nzGvHtUkNxDkYucNHeuCPW_Mqw5I4IS0dve_XCjayhv7uvFQTdinHTsHqf2ZtZAn_qz1ZnE04PEsE0KWGfHQUj6N5WIKcNhyQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,fTod1kMxyNrLe6wmpPVQB1vco-7IrNIs9Dx8h9kWYDCH7xYH8V5kj2iao_X4D62DX5S24Xi08LA-xy2zHwj-LIzSKOA38iBBAlHdMD7qu3lK&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,37VGZcLM1KBT3tttPbLrhK822id0s0HCTimMYb18Jk-8bCbWxvPl2N9_yFFLgXa5pC9DXbKX6c8DMHolUFme8KRWVQdBx_lbRwtpLIOp6O7uiuV0fttkzvI3jxdO&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,zIgfFXrrGxOB_lKTtscVc_jXQlq_pH7TMnaK0IkixM4MVqoSgc-Wp5VV-QGQWmUtW5dggcgtYpRa7OoRHCBvDew7fxYJq28eSZVguCiuBiH9DzdKTHAXC5o,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,b6YutPqRZaL3jtcBxfqj_SiU4M_x_Qx-GeSmBfL7c6CG24IUL7AdjJxpyosXZtCtKco4bSDRxOj7HJn5CWgbgbPzJ0TFuXzJjs7VRUKVhg,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,xdw7r1BqGJuJPUUyo2a9wj_y2KKTC-j8c0DAocpHKzpZZaVOgPcxdng0Xi_JMKvlGYmcUvlTfZMm2FYQ7pTrF20xWUreNaRQqtU6rlWB3kkuPmMNBALyTBs,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,4iCAaiDF74c7cCKTdUIIlT6Qu6BmbQItK-gaAIrQtpNdie21ccXNS_dCydALepUtQNTgsVsVUs1NavHOugg0IlFK0CjTj8TKnThm55ffUeOx8zHc4oZUYpE_LRsz&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,qBGyQwtcrkAjVKJZ9cZzaW1ivhGQgGfhpcJUS_ww-vfcQVNfD9OjDwDX1XjpzJWYpo08MqSOrKLpgO8d72TBVtSGg2OQ4AmggCv3IUgKeA,,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,Mm6fpsoFsFLUtanrkJ6cqlk2y1zCJbPDIfqxsd0pS_sfS05oTAHBNIXIUUd4iA8HJY0guikOejZmykugHqKeopxLvJE3UmpX9YX1Y3Gu2ToK-B8aRw,,&typo=1&ancr_add=1>)
2024-01-10T19:52:34.435 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 54, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.465 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 98, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.474 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 20, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.483 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 128, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.495 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 158, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.499 [Thread-4] AbstractProducerCreditsImpl.checkCredits:184 [] DEBUG - not asking for -1 credits on activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,BYjREAStKM-0e-e5OOY86CxL1g9WrAP01m5ROYtsGCMHTh3wox28vU8_r7Y9e7QwRiDtJ065Xst_zM3u0oFx_K7Fr-ZkN1GBxPR6aWEKrm4nQA,,&typo=1&ancr_add=1>
2024-01-10T19:52:34.499 [Thread-4] AbstractProducerCreditsImpl.afterAcquired:98 [] DEBUG - AfterAcquire 333 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,r4o5lwz6oA1VbUpicHhTmEpC9U7BwWBMw8ZYMcrl4g-jVFykL22Cb2WJIZLOqv-hLbxRqgThYlmO40Jo84JsPHe-IYBjUU8WMQ-jy1lZqxq3YQk,&typo=1&ancr_add=1>, pendingCredits=65203
2024-01-10T19:52:34.506 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 106, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.680 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 155, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.683 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 21, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.684 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 129, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.686 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 57, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.687 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 174, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.688 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 176, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.689 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 56, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.690 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 108, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.691 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 156, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 233, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 112, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.693 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 187, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.694 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 110, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.695 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 144, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.696 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 235, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.697 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 28, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 47, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 30, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.699 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 77, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.701 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 65, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.710 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 70, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 34, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 159, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.713 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 76, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.714 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 119, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.716 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 59, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.719 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 87, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.720 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 80, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 166, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 149, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.722 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 137, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.723 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 163, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.724 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 179, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.726 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 181, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 17, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 58, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.728 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 32, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 49, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 234, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.730 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 143, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.731 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 42, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 136, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 178, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.733 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 55, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.734 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 95, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.737 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 48, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 16, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 146, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.739 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 22, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.742 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 43, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.743 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 130, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.744 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 36, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.745 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 60, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.746 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 117, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.751 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 14, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.752 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 81, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.754 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 93, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 138, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 100, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.756 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 62, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.757 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 125, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.758 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 15, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 99, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 44, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.760 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 172, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.761 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 84, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 79, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 122, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.764 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 64, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.765 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 88, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.767 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 151, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.772 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.773 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 96, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.774 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 27, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.775 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 73, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.776 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 90, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.777 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 134, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.778 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 35, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.780 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 118, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 186, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 41, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.782 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 31, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 150, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 145, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.787 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 127, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.789 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 132, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 170, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 142, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.791 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 53, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.792 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 86, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.793 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 147, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 115, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 12, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.795 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 135, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.796 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 68, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.797 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 103, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.798 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 141, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 165, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 18, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.800 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 33, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.801 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 116, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.802 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 124, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 26, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 50, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.804 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 75, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 85, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 69, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 97, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 101, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.808 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 153, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.809 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 107, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.810 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 175, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.811 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 66, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.812 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 111, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 152, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 45, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.814 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 61, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.815 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 113, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.816 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 105, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 162, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 171, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.818 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 173, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 184, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.820 [Thread-15 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.afterAcquired:98 [aoy5vnfnjetz,6ek4wlg8c0ad] DEBUG - AfterAcquire 781 credits on address rpdm.v1.authentication, pendingCredits=63974
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.user.group.query
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.user.group.query, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.user.group.query
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.canvas.state.query
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.canvas.state.query, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.canvas.state.query
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:184 [z7wx0g93sdr7] DEBUG - not asking for -1 credits on acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.afterAcquired:98 [z7wx0g93sdr7] DEBUG - AfterAcquire 505 credits on address acf62840-864a-4eb4-a902-4861db94eb93, pendingCredits=65031
2024-01-10T19:52:34.824 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d43c72b6, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,dWlUyTFJQxRIqOG8poa5pxge5mYVJ48dUXfACv6LUtKsZzFSVGKyZagOScZPGCnB6cxsyzdobSyk8B6Bwd79504jcLncGDwqMeY-9K6dpOs,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ZNE9b4nkKfo7M-_v-eXIZ86i5mBDuVpXhcuhH7d4Z3TXAYIWBA7iz2F7WJKgkGmWz9vMjcTKd3WNKgAqq_OrNuwfMP9rqwnZZdfDvOywCznI2b71&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,ud4sg1bGnGSsxR72jGNWKmRRRqC9RhLjc5Fa_mAH3joB4qQCFRRYETYRnfp_427BSgD3-id4mG56z7G-3qi9hl8xOglhqXamz4f_kpydTW4Drw,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,5ea91cHeAd9MIKFvy9sbzhVk4n_LaL3aA10YRDmCls04v12YLb2XMmXw6eBUcLVuIE_Z644AmEt_e0ljvQnrS_I6HavZOLh4zjfiHWUvesg,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,jDru8ObAZFnt6UJuRwWa5fbG9hi1-FGT2jjgjYcb0j9MnKP4NE8WTbKBHSHyKvKnv60VBWqMlES8saZuop8zduWr_U-bg4uKdFl24xth0Wc08-CvUBpd&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,FCoOt1eEzJLmEhVMnIIHAKa-kTFceMfQD6Kcwc4ZSxQI7u6ROHuAf16piW1PXfz5fwuSX1g87RD5aUJh-U4htAPPBkPWT1OqrJG8TvqS_LoMCTdPQPVU&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,RozNWdaPDnLkxzHk0PIIu7yOI0mq7J36SJtiYqHJSj83AaF6otFCOMT6S0SrKP0i5_HdKBbJFOfyhgDON-XzQcdSOvC2Z4N90Agiv9ZXimQSY87tDA,,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3xUOFzb6MCi6u1j2LRQL6CE5BWbu5eM6Ir7EsfF2oCaYUBtqnA1r3VDYEmgBiVBb7QypDxbOzY9knH1MLZZ3oikFpY_p69_553LHOTbV7g,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,9xoBaA3kGi2K8zQ8BuNLDq0dYmsnE1nuJw9sP_cDKKpUiUl1eHBvRBJF127g2yN9LNqLqZkyPVSq8YusAWP6k6MCI4_4n846oL1LY84Tow2q00oOukNL4Q,,&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,OqfGI8b5WUrx4fzs6Vhc7lLcb55MI-F2H4j9ic-9Fbs1_FhBfTLzdobmjbVxUHuNV9kcphYOQdlHoTf8bN7hu4yIeKsdhwLW0rasAMuEm4mWhNV8t2c,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,ml--e7XB5_TgtwWFh6lw23pXHYNiKpjFGKoUKmwrOwHyS3lnFQ78O6WUjq-OM1ITxXTrgD6P5K2E89TnqOgGcaOyn6aLaVIcvPhARMyQEqlFDI2mcnYwIfwADKB-&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,olHwwJjC2TUJmGxt3e1EpgRMfVMAAdmudKrqZ5ZByilJc51VAK_ZZHGL1mLqUqv59mtQqrIVOT0FHxcuVGnfmTsHrtP_Lfh6vEjgZmLr0Aa8EqAKSU0BDQ,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,3crdMkhGhlkrJXp-HsjZIEJOjIoajfvvI_g-uTOHgEc26-rm5spEwaDR4tTrIOKeN5HSQz3HfKcnanjUMfuquaMfvk7M7K2AtlUVxZdR9Q,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,Qv7RvLuSjcsV2_72qHHLM6lmOI05RppEGGyR8280SVuKWQgOAl-EeFo8ZDeB_9Hw2f5ZgHbKDJbuB11rT-FBX6wf71ASJsIGNZYXzHn1HnNPy3pXaBaEgvrJvA,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,7hPEKeOrrW0p5qUGMytIu7avFNFhxNhXFzKgo6a-hDqm4ky01dphgxAxTpF5DmP3oUrVV1uWYwMbJuUFv1IoX_o9-H7Y6xXWdj067c1A&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,dU7lk32qJcXeW-cUtM1E65BB47JiLhSgJYTqhLIX373Pj0miueYi3kN7MYtAU5zT39qOtzzAiHcBVYWsGDB3N8R31seI_xbm-rQzfd0jzb6V1rSLy1kotiI,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,BtR0iFYtXaxmAlygBDvC74BLAVELzJivsFTs_YS41zS_XagPPAr0UUKW1lNrafWE3WxaZcW_S5WbNn583_uHQ7aLAA9vskUJfxaMwUrM_DxAOd2N&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,nW_o_ayhGlEOytZThvhNsFjqKDq7oEaYAEjw_jVKEHAnXglYBlSGSA3yj7wyG-2DXRXiRZSD11UwsKx7fSGPuHkUUSoULIYqacVjSrfmUuEfG_A,&typo=1&ancr_add=1>)
2024-01-10T19:52:34.824 [Thread-19] JmsStaticConnectionPool.onException:78 [] ERROR - Receive error occurred. javax.jms.JMSException: ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:714<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a714&c=E,1,3FhMyZ8H2FihvFzilmw5MqmH_tkqp5vHbCOJsN5x6sQNSLyFELuJt_Ej2M_24CH42Xx0iQesALGDJK8HHdR_PDaLploHa8R84-uwfftA3DUM&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:735<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a735&c=E,1,JvpZWdXDc673FDsIBWz8WgGu29y6j8H0zAczvf_UXYyUAzEMKtvXJ3tMkleaFsrHU-QdlAUi0bWddjnikuurWEEMfZs0jIDWZ3jnrx2__Vbg93yZ8oo,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:868<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a868&c=E,1,AznnL1cSXDTbCj6iQ4ovEeP71-oopbCcH0laxV9dB-Y_kBuerNfeOAXRkRLybAqgqQBjJZKBNFEoveCJjTGncMCguHnY0hvsdZuKIHz6BEhvRX7N&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:856<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a856&c=E,1,vBewrTh4rfp_ezwl572jqeiy0VBShHwQJNRT99yv4B54VM5Qaf2hTHrOq33emXwk_WSFrcueJSCKI90AxcN4jUznfP1ZLzL1dJdUz01zK4vN&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:802<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a802&c=E,1,EY-ry2DzdJloXWtaemW-dWnRy1-rr2mQFFjoCbmWjX6xHCNZF95ldY2zUgfhOhSQFfzm7eYV0PdRUTOwJrx9jCD4U-B5ztFqWmvqEmJQuqdCyQgypA,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,gHouFFMo7plejV1zi4ureD9QEDkW-WjvytRlH6DOyEt7oMO8ULkj6APov-gDhxtXAspTlk7qF6RYw-RrCqfSrXlcE0Cm45J61qJHt5Yc3zuBJJyEX8A,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,17MGSyeFWTwZKhb2DCqWeyFvdlinlKAunc4JSHJltjjiaCeT_1UjXFsTMtEiF_NYIOMI3brJbPkRW1QkJIhe_Br44jy1od8-BI3l37IfFJzZbcculrc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,F6NyfiAiJI9fxHnOQnnpO7dLyoFavqwnNivvLL6x2LO54G8SpNhXjlUN748SkNWwe_oCD9JhI7u_BOk1uNbwSTYV0lXRyewBrwvvysSEOSCggiFCFRgjtA,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,TnFdrKl6nCZv9PR6lSsRZ2YN0r8LqpGQMRo3oHlgyE7_AgUMzrW12R4HeB5JhX8Akv-1iKeVGrnTLV2WeAbBOCM-fukhfRPsdUgq5pwe0yU,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,vAtPtyOuzaufOELKjMWnOBMrJEOShH0EzH9G3WL8gak4hWHmnVWPL1W7c_YnxRH2IerBQErg9VJApOiiX6TR6tcPau3xqRHrPyrsH70KHg2v&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,pak9EX0xrMkRTN71voUljZIOmUvksqdCCqWdwh8fq2R4OZXUQFIHNH3P9j0t-wpQStfSs_ZsT0a9BZYopaoyMSnMkN3xqvXn8A9dTehv&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,acJEflBFObRkFNBskLYk7AARfS1_CzZH3yFGYB9r_Ype7n0DAWduZhGpmImVc9k1pTnmFoA8L8UukMVvWBinxujU6r75ipmzh4I_cNQ5Gzmd97g5SAU4dRsf3Hw,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,aM5tVNMvOtLUxuoV8Dez1Y7fAS8KwME3peX3IqP_QPViIWgkZQahJW_1BMBQUo35u5h9hGJ3XSyDgkECwWOGDn6a9W-gctjh8WvUKx9yQ0sSLwheMcs5mhR2yVM,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,fUWcgLX3hupjGCjPYEByZ8xfzFx4iN-VnkLB9MarNu6OEOfrwnCkvJvtu6e-ZV-FREp0Ssisi_fezDu18eP9adS4mmcPPlJH69vnf0YDVg,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,KuAjbB9H9ddyi6unfXrVdNXylpB7l6qLIUfJARScyUnc7iH1GtvO0s1MAtB9fwkdqXq8idOJYZs_rg0iU-sN4aV24ID0Os5tsyvB34ba3ve2D3diROqixzkQuw4,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,EcYd8WT9GtMN9IXo58gsQpj5uNwcdCRtDfa0zG9t2jpvfdaSX5r9vLey2icJ-4wo3EyBMyUQzwyqYwC5BMfe4yHGjyo7h8jxmjcEKig4V0KyVa_SVq58NBT8&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,_a7OW7NorEoCm9eOnYjCi6TFl2__or_ickK-DwP4rrMmRX2Rj_TKaxpGjSAwCJZasllt1gefhShoob68MthMyp6bxv9N5fucG2HiF0g3yK4kqHTZjiFn7kxZ&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,-ttWHQzQDZ94E-HqKtYTKV2nuApkTk60Zy25iWFIykqfNe0o2seEul4UE_gETZr08B40bJjmZKMHq_-R0M8bQ8Fbu8twSkWx9MPLFUhc&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,624T4PuCyLmsS-UG0IoXRx3MMz4DqnU_1BuWfOz973WSpqfLsFFDAICu75KCoCwAs9O8WgKTEcBVqRhhf1JUqI3Znxem3LsG7Ml0Td3DnJJUoGYR0BE,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7UL50c_ahB0IhSHbe9OlGABBKKE4OFyNM67SjmNrSo_LQhfViHAlLgvwdlRpbmDdymI5VaeOEklHjvlweLBz8Ske6wjc8J3CzM05VYenh6DzNiI,&typo=1&ancr_add=1>)
Caused by: ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
... 7 more
2024-01-10T19:52:34.828 [Thread-17 (ActiveMQ-client-global-threads)] ClientSessionImpl.acknowledge:1225 [] DEBUG - client ack messageID = 10054445
2024-01-10T19:52:34.862 [Thread-24 (ActiveMQ-client-global-threads)] JmsServerBase.logRequest:117 [] INFO - REQUEST IN:
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,01YVdf0IHIl4w2xIwUh5i7ToRiQfv6TxT-FRWSNW0n6e2aucHkWVyKxkYJTpG1jt7_EUCuQVS0NWGalDyf11AUIb3qVPj67IUtjJB484IdLKrHgbmSd-Q8Vg7AKZ&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Tuesday, January 9, 2024 1:57 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Both the "replicated-failback-static" and "replicated-failback" examples run without issue out of the box on my machine so I'm not sure what's going on with that. One thing that's odd is that out-of-the-box the line 122 [1] doesn't match the method call in your NPE. Perhaps you modified the example?
Aside from that there were a handful of questions in my previous message you didn't address. I think the questions about thread dumps are the most pressing given that you indicated you're deleting the pod. My guess is that the client is getting hung in a network call that doesn't want to return due to the pod disappearing. A series of thread dumps during the minute delay would confirm or deny that.
Justin
[1] https://github.com/apache/activemq-artemis/blob/2.31.2/examples/features/ha/replicated-failback-static/src/main/java/org/apache/activemq/artemis/jms/example/ReplicatedFailbackStaticExample.java#L122
On Tue, Jan 9, 2024 at 12:47 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hey Justing, thanks for the help.
We are using version 2.31.2, which is pretty recent, if not the latest.
With regards to the log traces, other AMQ threads show similar behavior, and our system isn’t doing much until AMQ-client stabilizes.
I am running on a k3d cluster and the failover is triggered by me deleting the dm-activemq-live pod, which k8s then spins back up.
Also, our slave config does NOT specify “allow-failback=true”, so the spin up of the new dm-activemq-live instance does not trigger a fail-back, which is intentional to eliminate failover-failback chaos in such a short time.
I am trying to setup a simpler test case by using one of the examples, artemis/2.31.2/examples/features/ha/replicated-failback-static, however, I am getting this NPE on line 112:
Caused by: java.lang.NullPointerException: Cannot invoke "javax.jms.TextMessage.getText()" because "message0" is null
The NPE seems to indicate that, you cannot call consumer.receive() again even if NOT acknowledged, which seems to contradict the comments on the code. I have run into this same behavior in another example as well, artemis/2.31.2/examples/features/ha/replicated-failback.
Can you please confirm the expected behavior of the example code in the 2.31.2 version? I can then modify the code accordingly. Or please point me to a working sample code.
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,lIOPl76F2R4EtenIAEJMeQ17kfmHQ6buC7G1W5TRefOcSfxvj_-3dLXV3MKDkCTkac_65Rw0SYmof66i91zGwgb_SS1nKaBpao_Lw7JNNk1mkE33&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Monday, January 8, 2024 9:20 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> Now, the backup process does start processing msgs right away...
What evidence do you have of that?
> ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable.
The log snippets you provided show a reconnection attempt back to dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to dm-activemq-backup-svc:61617 both of which fail almost immediately. Then right after that another connection attempt is made to dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e. from 20:03:44.874 to 20:04:44.933). Are there any other logs during this minute? Did you happen to get any thread dumps during this time? If so, what did they show? If not, could you? What was happening with dm-activemq-live-svc during this time? Perhaps there was something weird going on with the network related to whatever you were doing with that machine to trigger failover.
You might consider simplifying your test-case and then adding complexity bit by bit to see what triggers the difference in behavior. For example, run your test with the client and both brokers on the *same machine*. If that works then move just the client to a remote machine. If that works then test the client just reconnecting to the primary broker upon restart (i.e. no failover). Then finally put all three entities on separate machines and test failover. Hopefully such an exercise will give you a better understanding of where the problem might lie.
Lastly, please indicate what version of ActiveMQ Artemis you're using.
Justin
On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
I’m working with John Lilley on getting an ActiveMQ HA configuration operational. Our current HA config is replication with master and slave instances and with static connections, and it seems to be working, except that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect itself after a failover from the live to the backup instance. Now, the backup process does start processing msgs right away, but the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable. I have included the relevant log traces from the Thread-0 ActiveMQ-client showing the Connection failure detection all the way through when the client is usable again, over 1 minute later. I am hoping that, based on the trace, you can find something that, perhaps, we should be doing differently in the client side?
Here is our Client connection url:
(tcp://dm-activemq-live-svc: 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
Should we also be including the corresponding connector names?
In an earlier email with John Lilley you recommended 2 brokers using “shared storage” instead of “replication”, which I will try next, however, since this issue seems to be on the client-side, I want to make sure our client-side is setup correctly.
Thanks in advance for your help.
Lino
Thread-0 log traces:
2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
---
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616>
2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617<http://10.43.11.76:61617>
2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
---
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616>
2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = ced14fbc
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617<http://10.43.11.76:61617>
2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 10, performing as a failover operation now and recreating objects
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,hqiYs6vnJzfq-X42G9W9ohJtYgn9LRPR-82RHxMyPnKUD5xBwCbo8W6b-PegfmAGwvEENlDO0k2IkoGFFegGxR-H6J72UHHlpEFbGtJzhCF3sucQ&typo=1&ancr_add=1>
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,KX9atZvl9XxpGPWobW9hXsmaLrVekfw2221zvD0PJ9Ik3fEQVyHKcZZCe7MQRciamxD-T-WDEk5Xa7_cAt0PSHwVW3jG7zzhq8XH9XIRC8EXmg,,&typo=1&ancr_add=1>, needed = 65536, arriving = 0
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,ZOCwWoQv34cYLuw9hZvJzBHfSrX6oQeQ_3-SetFMJa1hRC6lXgiKsAElOhfZ_vfnBhRGRJPZjSOqHIbwWO2ltSYYY7ElQjyItp0uDo79_jaDkw,,&typo=1&ancr_add=1>
2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,obbNTaFqn6mglgyZXy1S-0nBLNVtEYlkZBwAEKSGDGN0RuqlR_-7fIQPW4ELNqVFJdenAcKh65C3yQ2b_Lef7W01Tst6ZTrmI66KfHLae572jXv-HF3O&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ytVVTXzfq-J1NXl2ct7T_47F5EbfZNQTXP8dlqw720EiCe32c6CLwlpk0-v4MzUD61FErq0nkRTalFpcTtIN8TDeBvz70rzb__TyeIDm48ZKfFU28qQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,KQYUuEgOaR6V8a_ZC_xBpKZmDneB4LTL1cQzn2Gqtla8ntHJ1fgRs9KelLgvbMd9zvBTg8maXUerYZYQZa-DlfK7fon0JZ0O74PNJ1SOCHqIPN4fZQ6Xv731OGOD&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,OSVh6NSORxz7qZ6eIuUDgWskoYfyXT4NNdjJYGMbH7HeWEUiymys2DnptjYKdmQfqjfHmI-egRZ31yWwd9MKzEd-gbbsuTUsqn_46niKaw,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,DM2boEBGg44zzcd0m37ITkOnfDrkqLE8ZSFGJpEI4AGo2GyOI2yKX08AEkKHxw_UnDMF1vccYEvESFDXjXBBjjIGO5TEsMLWZHbJF3AOIEQ73uc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,wipgGeA7UVZIy1uuqvuxJEx4BA8TvYfSYmO617IgrvnOo-z7sn1vUjXFqIEU9M1HmedBVoHxB_LpvuEZ9p_biioPM8V-V9eS8i8oST46GvBhZZIc&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,0FNTjscT_d2CNIid0J8d_x7C7Peb6_neE-a_MZeKqkM1o_sPGujvpkXuiLv6kms9wSFyYqSux86C1MX2R8WKM8IZBsjKt7sCs03U2lRYlVbw2eTDg0ihSEV1nA,,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3yKPqxpIi9n43QJSCQiysKGlEYttZ5Dmuirc7jUAWNz6orLfIUCYwdDEJ3eU0QiHywsNAJdWqrvnx5rZK6-8TTJb6o8tr1WSvSL7omLF&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,6B4qLrj7q9DzDb0Trp9xgst7cOBhbD4hcEcGyPFul5E_mh3_oGmColB9YQkQTB1gJ6x5Mcm0VwvBLddpDpfFkcZwlzb4YNR9Ym5fcDclFW97pIrQ9oG4&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KqR93A_gEw1FpU7__PsYjSkOC5LH0zFTSKu5K9YlCdVAz3h_c9EkBndO7aliE4RSbK_qslEZp0b-YzJfCOxmB22jAfplsYxguMl1qjC0rWpNjfr9_fh4RjNL3AOK&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,mhChlKkihYjDw_Kk5SL1d_4eb7zhBlSzri_nwjOUsrs6MAvEfDAAjia07yTMUU6CQqc0WbDS2Fn8HJJI0UwJh1N7SKiz9dsP1gJLmgY66huLhXZE9c-3ECGXyIWQ&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,nn8IDmg6XQJLLXzH7kunXDNZa_3A7fUlnHdRCLoha7gQWJO1QeY32duGFAQMYaFHsfIbHbv142mei8HfCqRwrpuUfikYxLkFoXaD3pORZb9roBJs0H8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,DPLEvt-MPmmzv78LEkeXtYHM2q37m12TxX-nqjuIJurQdeXydsSakco9t_O7jgBbQ6y115xxdDb8IonFZbM86zgRDLcvUmeOSJuDmZybD_6JaTX0zpbQ&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,1vNFV-mDaLK92lJo0Mq6jvkCDzuwrSHT4lp1nB6KTzOCSO6comH5xNpEcvtZ_9pc0cQCD2kgDfB5xdxg0tAX2dF5oG5DHY6kxWXOBLnuAwiz1BkEFhw,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,aLydz0SGsclotqCro2upcTg44B_Ij7ckjApLu4B5JUwZKKdMo5z3RyUR4ikHFY8L3t_yB87Zp5atGRwoAPA1L9Mc9c681UjFrG16n-udVjeHtxQNxx8V-Nug&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1mZX-u-XHEZ_e4Ut7YETRvWm7U_qkC-q9QaPeHah4K4yNl5foyM1VSExCtYiisknkelDj6_K2MUHGc65Kd-6-JG_TXLby0OPWodJVjX3BtKaDNBUzusgpzT6ajL7&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-nbEPDxI7J-DFvbJfVVt2Cs1Vi_hN5bw-8vQF3SlM1nthw3VbJLWmS1h73cupUxqBoHCImTzDqI1m5gutx3ZriLpLpU7IqutHCqZuT6CTVWACPtj&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,-lUcdhDzaHWdAWLCc0IxWCFkKtKHxoEd64GWhzXAbeuG_Zm9mvn624BY6uS8MgQn-s2TMF8PbEPYVbOOy6BJuI8ByeiuAF9Y8f2I5wib&typo=1&ancr_add=1>)
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,T2UV-1dji1RODRcU4Ff4j_IzfTH2n47BatzyLZcEOU8VROFr3GS8re7CjDcnnHCSaPEtR-35J3z30dnYcz5K66rVsMVxgDiyh0FvEDKnN67naHjYbn0NpzAc&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
RE: Looking for HA/replication boilerplate broker.xml and advice (continued)
Posted by Lino Pereira <li...@redpointglobal.com.INVALID>.
Hey Justin,
So I am trying to verify our AMQ docker images in the local environment, basically our system runs as simple processes on my local Linux system, but the AMQ instances run in docker containers. Now, this works fine for standalone, and for the HA case, the “master” instance works fine with, or without, the “backup” instance running. However, the “backup” instance doesn’t seem to ever start up fully, that is, it is missing the AMQ221080, AMQ221003, AMQ221007 and AMQ221001 log lines, shown below from the “master” instance. As a result, it never announces itself in “Backup” mode as seen when run from k8s. I am using the same broker configs that run fine when AMQ instances are run natively (no docker).
Also, when I stop the “master” instance, the AMQ-client reports these exceptions and never connects. Any ideas on why this may be happening for the backup, which uses the same exact docker image, just with a different broker.xml?
Is there some fundamental thing that I am missing when running 2 AMQ instances from docker containers?
My goal is to validate that our AMQ docker images are functional outside of k8s.
With regards, to getting you the thread dumps, next on the chopping block, can I just attach them to the email, or is there an alternate path to get those to you? Apologize for the dumb questions, but I am new at this.
Thanks.
Lino
AMQ-client exceptions after killing the “master” instance:
2024-01-11T15:47:15.322 [Thread-0 (ActiveMQ-client-global-threads)] client.errorCreatingNettyConnection:741 [] ERROR - AMQ214016: Failed to create netty connection java.lang.IllegalStateException: No ActiveMQChannelHandler has been found while connecting to localhost/127.0.0.1:61617 from Channel with id = 0bec947c
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:980)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:866)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:848)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(ClientSessionFactoryImpl.java:1212)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1333)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1253)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(ClientSessionFactoryImpl.java:1496)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1074)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:959)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:697)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:559)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionException(ClientSessionFactoryImpl.java:425)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$2.run(NettyConnector.java:1252)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
-------------
Log excerpt from AMQ docker “master” instance, showing lines missing from backup:
2024-01-11 22:38:54,844 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST]
2024-01-11 22:38:54,853 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2024-01-11 22:38:54,905 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2024-01-11 22:38:54,906 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 0.0.0.0:61616 for protocols [CORE,MQTT,AMQP,HORNETQ,STOMP,OPENWIRE]
2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now live
2024-01-11 22:38:55,114 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.31.2 [live, nodeID=320ff011-b0d2-11ee-a54c-0242ac110004]
[rg] <https://www.redpointglobal.com/>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>
Sent: Wednesday, January 10, 2024 6:57 PM
To: users@activemq.apache.org
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> ...I’ll have to instrument the code for that, right? Or am I missing an easier way to generate those on-demand?
I'm not clear on why you'd need to instrument the code to get thread dumps. You can generate thread dumps through a number of different commands, e.g.:
- jstack
- jcmd
- kill -3
You can execute these administratively or just set up a script that generates them every so often.
Justin
On Wed, Jan 10, 2024 at 7:43 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hey Justin,
I did finally get a chance to run everything locally, and it all works smoothly with failovers in both directions and the AMQ-client recovers within 2-3 seconds and with a lot less log noise.
I did not get the thread dumps yet because I’ll have to instrument the code for that, right? Or am I missing an easier way to generate those on-demand? This is happening in a k8s pod, so not easy to connect with jconsole.
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,P1-JTENl1pkpwfayJeVxTcbmyxoXqv5aWc8dgoVoQa2P9YJ6ITYanSwPPWI3OOvd-1ZuyvCfDdUgZSHNKSzXPNMT51ZJGff0MJI1JNYM4KnicxOXm0JmVYcKXz0s&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Wednesday, January 10, 2024 3:02 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> So our issue seems to be only for failover from live-to-backup, consistently.
As far as I know the client doesn't make any distinction between these two cases so I'm guessing there's some kind of environmental difference that's making an impact.
> Is there something that we are perhaps not doing on the client?
I don't think so. The connection URL you provided previously looks good.
> I will extend one of the example codes to use threads and sone of the AMQ-client wrapper code in an effort to isolate this behavior on a simpler code base, as you recommented.
Before going down that road it might be better to inspect the ostensibly low-hanging fruit first. Did you happen to test everything locally? If so, what happened? Did you acquire any thread dumps during the delay as previously requested?
Justin
On Wed, Jan 10, 2024 at 3:04 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hey Justin,
After starting fresh with version 2.31.2, the examples work out of the box for me as well.
And, after ensuring that version 2.31.2 is running everywhere, client was still using 2.30.0, and refactoring to using the recommended shared-store HA, our AMQ-client is still behaving as before. That is, failover from live-to-backup, client recovery takes over 1 minute, and logs report what I posted earlier. However, failover from backup-to-live, client recovers within 5 seconds, which is the expected behavior, and our system is happy. Please see new log excerpts with more logging included for the 2 relevant threads.
So our issue seems to be only for failover from live-to-backup, consistently. Is there something that we are perhaps not doing on the client?
I will extend one of the example codes to use threads and sone of the AMQ-client wrapper code in an effort to isolate this behavior on a simpler code base, as you recommented.
In the meantime, I thought I’d ping you in case you can think of something obvious that we might have missed, especially since we just moved from 2.30.0.
Thanks again,
Lino
Our versions:
ARTEMIS_VERSION=2.31.2
JMX_VERSION=0.18.0
Logs from failover from backup-to-live:
2024-01-10T19:52:29.588 [Thread-24 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d7ff7885, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,O0LJ4VfRCemWEaMLULajHsgUo6YseAVDyBlrTm2G50vM20WhlTUV0ueY1DxMRAMStXAoc6wy5Xx3x1vi2XsnE5F7_iNbL-MZ03M4IS9G&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,VdcejKqF9B67eWPDv-PP_0gR6PEmUxFYHHtW8kyn3Gzg3q-HHzL-Lr7oztjj2LQumPJWO-1KF4OPavU8c3DW4zPgrUsidhnnCpoMUUn_&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,xKCDuAN84vJBjsCp75epCz7V1lR05TFhETdvpPcKntBTPnQk0zuuQSei7-ly1RPUUsHDP8ZAe-IevqwGtTGfEB_yNzw90QcOpEx5BaXOW6E,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,pq7BenEb7b9i-FYz1LnTy-gHYDdV-28FiSv3pkV_i-1TQZtRxakbT-rOKkgLgz2IDx875Dk5EE_OrwsUcew9TIsvrcZZZ9ZIlwboxMEq9xnR5odpQL8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,8mmrZ5alRiVlyEshqjlGpedpt7TuUVBJOy8DDm2qHLGoY6v3pR76llV59sMsYwy2v4rwLqiD9Rcf3Hg4QSCZo7rdO1rQgMNapCKji2jg7T4IJOk3OAeLnIs,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,RLTYzCoyknTJ1GB9szMq3Jg02QAnt7juedv1HCTYmmA4JU1DO6MlJhkvzOTBqHbeeOdKSvwkualBrhGt0fUk7Nl5YLzlgfepqTWcDiBmXQ-_xSMcAHPw&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,iYtEGEWoGf58Vq3CGAxBrH3iWZ1jWm1N1b3AOFPNlx0kCahcxHEwXT7ZAQ_h3ynXTeKJSkvmBiGUkbqE3bH7tGw-Gn8VV7R7LaCkxl9Dch5I4kQDfJDKSg,,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,kH54RADAl8RcHDeCxJW9b-P4YJJFElj_Nbh6mHtmwlFdP2tKzzbK3wkAXg9tVqhIlsbd4gr2LJkn7DWZ4I-J8WVE-7UovVbVakuQTTxc2pbR9vuIDiEFztc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,V9-XUYgB7GoqPyGhEdE_CmtNood-yvXw6P9wRIqQOSYSWuTfiF1bhonQejBQfgJdHq8k2TsE1RAvcSOp_WzBE-ybl0l6rc5pIuMGdC1LUITpdk_Iuha7urSje4Y,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,RNgnFH7dBkD27TX353Mc-kqneOLcqFXEgpGywxg8VduTTYNyYFt5VqkKfirozQw9V88WOI8XK4v9dvrtPIl-3u7cp673GvKxq3L-XgqypuDtSnkRIYnh813U&typo=1&ancr_add=1>)
2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d43c72b6, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,L97BTn6AFLbUpYgHzAlQm9pDJ4Sn--TlGDpz05xOE3xd-qn9krfLJ6JzGTaiSpQwuVjuQ8KUPfUSW6N18pgw44-ay7eqYb2h0p3sxehZeXAC&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,PiedcDlDv-nxD-EF13k9j-sagD4clTOVvCs6xyDQvnrPzDUxc4xAbnHg0d7hnqeG4cVJZaO42LJvoNm59E3SmClgLAft-3N7F898HO6Obbkm&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,G5SZ7tGYpGBQzNRxdmIzsp4hWlUsmbY3_881k0N5lFoFBwuHt1dMCkKHI8wiZRGpS6qDY_UDBvDmKVQNRIHXIIyg5Df2nnWDCTLnbvWCO20fW0PC&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,MsagygznR7ie9qgJLlMogXq8ex-X4ST-fg_vbVbgcBanXur80DVykS1ftnzq0e4bWLE0k8EiNmlhlezemqTlBEncWMJz-AQAKdTxEk4WyVY,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,n7l9CUpZZbFH3ca7Iu2wxM3kNfged80mMRcu0Vn9wIoZz7aKzCP3c1OVR73iqF0oLbir7CjMHKAu3sfrR2cLs-EP7orqDWQKQApkzTZra9CJag,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,PePvk640nUsvwJWNJY39J2R954rSgiFVPPSQLOsjYskMxtDnDVBuzQiZ8ut5SAPoqzV9uYW6GX3ekOAAoou9eu6VpHa_oFc2-7d27W00VBLU_o_Zv4wEUw,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,REmfZe55ytZRsqZvFkCNsYUcRO5Yd8PZwIlQfGystV0qUeF2mXSYaWKgleI098Ac40PND4ilF-ukqEWDY8N04AIRQgGgKGpG036xx7jNSc78YS6UbKWiMNUHfNc,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,nnqfN76WKNs0ZYhVPzea17QO5aWkwmLqbfYrmQJ_qGQIgTzCL4PSXhjU9jy4mhbCUfl53cwF4ISYabr_zD1GUvCgv5m4dYW1_OROCBpe&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,vlJ5vYZAfrlVp9kL50a_jwWhqVW8-LMUgzeBb6mn-udcS_fMR1yld889bXgOXTMDiutFqKYX9XPh0RtxeXRLtnN31WqeOlH_CZv4_ZyhnOwXvm4J&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7Ba-VAkMm3x0E0f6F1f73TH_y9Ylo03hx9DEz4kqWzvrXy0gAv3D9zIM35eXIaB95UFxo72DkMpb0nKW_a-BIF89JobbRqjyVNLdC5K3L1EM520B&typo=1&ancr_add=1>)
2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:29.592 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:29.594 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:29.593 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:29.597 [Thread-6 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d4cf19d5
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.600 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 7b2392ac
2024-01-10T19:52:29.602 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.602 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.603 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = c1fb4c82
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.610 [Thread-8 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 06b7b3a3
2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:29.611 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 65680be4
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.624 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 97a109b7
2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.625 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.629 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 77b2de2f
2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:30.612 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = f92d7cd6
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.613 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 1d44895d
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.614 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = e8602acd
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:30.631 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 0a217bfd
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.632 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d2aa00d7
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.633 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 9a4f639e
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.634 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = ecdb1cbf
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:31.000 [Schedule-Runner] SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service as system
2024-01-10T19:52:32.001 [Schedule-Runner] SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service as system
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 2/200
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:32.116 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = a6b3d488
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.117 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = c0627cd9
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.118 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 71523eb3
2024-01-10T19:52:32.119 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.120 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:32.134 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 2/200
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:32.135 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 68efd0dc
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.137 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = daddef73
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.137 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 84813ce3
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.138 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 533990e9
2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:34.370 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 3/200
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:34.371 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 1d8df71b
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:34.373 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = de20e193
2024-01-10T19:52:34.373 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 3/200
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:34.390 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:34.390 [Thread-10 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 4661fd5d
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:34.392 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d1c14a74
2024-01-10T19:52:34.392 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.404 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-10T19:52:34.405 [Thread-9 (ActiveMQ-client-netty-threads)] ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up backup config = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc for live = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.405 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 38, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.409 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-10T19:52:34.410 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 37, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.411 [Thread-24 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 10, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.411 [Thread-11 (ActiveMQ-client-netty-threads)] ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up backup config = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc for live = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.413 [Thread-24 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,S5999ok8Om6JjNfBLhC0R2hYdl3dGqPWsLpis0MTx38AhKXPNqMMlH41RrDz0UZsxZzSZbLvUEOYFEa7hwM4N7gAZfCS9udcqSmOwhLg&typo=1&ancr_add=1>
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,CF5ZPcR0aBH6uaRjLjSeWoBKLnRwUOOgvrD5MzWuerKGIs1FEa5vZF8rdF1kWSKbLwGQlm1IsOeyey0f1t7eiA1ApUgs1EDhL7ilSV0c6rFxEanucXI,&typo=1&ancr_add=1>, needed = 65536, arriving = 0
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,nl9VlpvL5Lx7cxWUON3eMxxXV1my7fPq5WQLaWD1ozszhEPr1wVfqCwFilScj02MzRACloIhLZ8CgyMWoa_OqCu-Tf4folOdvKvtooRq&typo=1&ancr_add=1>
2024-01-10T19:52:34.415 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d7ff7885, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,awSP7U_0_MIm7Km9lwe59SrlpKsd9iUMA2y266CCeNZS6GJuEExdDYVS6EVMnlmPlVN0W1aqSx2Rss2Dm_YIM71s5j7OAUz6pAdGaTqnU_h2RyaoD6KyuoU,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,8gAKfG0MZn0OMdhCHJ98evOezuGoCS0mNbGWplSwawxhF0ed3tPq9vKWoWUbn2xmoLJ34skkCTpI2tUuevSCeVNT00tNpFlOJlyAjGWu&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,tXm2hjddgUkJac0QpMg4nVt2gwg5e3ORdql8K6xsJlBZIxph-NuPt1Ks2iB0ZarzoFELPAuAerVMiQpEU3nkq1JiqLAju26hE6Y2c3MwmKRTj66O8tsfQ-WFSh4d&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,P0xsXGcezpL-KzlLSOt_rfr6b6RF51B2hthJzXMGD04QeyLyg6AnMvF4M8ZOEwF90ojRdT3CtOy0x9HBSpGK_mmF_6Zl7Muz2TKrZRLAxQRxhV5XnvYlwQ,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,qctWvKkM1LAyegiROEl3hsiEN3dPcBcH4jlkBConOP-G2cgGl6X6XG-kdeQuBBOC8BmtTf-VmlbpJTjC_wznn8I6g7pLOg1uof2v0j_r3yyK7U8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,KRtNFryEe6jhqdT3of2FZC9PF7zGlYTxJ3KPFwO6TViZkyaHiybwHP3JmOhQ4F7yZtorUVRe5phyCswDWLc8lIFBVowIJygjwxLmO-F3xJSonSQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,ltH-LxKKSiKev4cs4CgS53B1Sn1z2t9IkxY_vjCKeoXMalJfF0yTujWPogG1a9qkWV5DrAT_qCTk7MdQa3LpkAae5yHTe_19fREF16p6vMMJqGXqLjw,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,66enqLb5-TVz_cB0ZCkDW4SpX4c3v0jslINMsxhHSjXfBWrIFoM8PxfSS7_VWmc_MdfmjPHIKcLm4zfcCe4MggwEWLxB-RjiUMqnmc8ShDJLoit5&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,xM_aKOMjg39ORE7gTvaGKP72ytDNvt6tjanfNTzekduYVGBvn963oijBiSY1JOQeorfgcCf57lVZSYIQ20x8sOFAH5zNexuTI0rBtrkDrtcJpq-S&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,Jr0OyBaxbJX-r7nzGvHtUkNxDkYucNHeuCPW_Mqw5I4IS0dve_XCjayhv7uvFQTdinHTsHqf2ZtZAn_qz1ZnE04PEsE0KWGfHQUj6N5WIKcNhyQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,fTod1kMxyNrLe6wmpPVQB1vco-7IrNIs9Dx8h9kWYDCH7xYH8V5kj2iao_X4D62DX5S24Xi08LA-xy2zHwj-LIzSKOA38iBBAlHdMD7qu3lK&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,37VGZcLM1KBT3tttPbLrhK822id0s0HCTimMYb18Jk-8bCbWxvPl2N9_yFFLgXa5pC9DXbKX6c8DMHolUFme8KRWVQdBx_lbRwtpLIOp6O7uiuV0fttkzvI3jxdO&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,zIgfFXrrGxOB_lKTtscVc_jXQlq_pH7TMnaK0IkixM4MVqoSgc-Wp5VV-QGQWmUtW5dggcgtYpRa7OoRHCBvDew7fxYJq28eSZVguCiuBiH9DzdKTHAXC5o,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,b6YutPqRZaL3jtcBxfqj_SiU4M_x_Qx-GeSmBfL7c6CG24IUL7AdjJxpyosXZtCtKco4bSDRxOj7HJn5CWgbgbPzJ0TFuXzJjs7VRUKVhg,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,xdw7r1BqGJuJPUUyo2a9wj_y2KKTC-j8c0DAocpHKzpZZaVOgPcxdng0Xi_JMKvlGYmcUvlTfZMm2FYQ7pTrF20xWUreNaRQqtU6rlWB3kkuPmMNBALyTBs,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,4iCAaiDF74c7cCKTdUIIlT6Qu6BmbQItK-gaAIrQtpNdie21ccXNS_dCydALepUtQNTgsVsVUs1NavHOugg0IlFK0CjTj8TKnThm55ffUeOx8zHc4oZUYpE_LRsz&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,qBGyQwtcrkAjVKJZ9cZzaW1ivhGQgGfhpcJUS_ww-vfcQVNfD9OjDwDX1XjpzJWYpo08MqSOrKLpgO8d72TBVtSGg2OQ4AmggCv3IUgKeA,,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,Mm6fpsoFsFLUtanrkJ6cqlk2y1zCJbPDIfqxsd0pS_sfS05oTAHBNIXIUUd4iA8HJY0guikOejZmykugHqKeopxLvJE3UmpX9YX1Y3Gu2ToK-B8aRw,,&typo=1&ancr_add=1>)
2024-01-10T19:52:34.435 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 54, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.465 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 98, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.474 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 20, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.483 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 128, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.495 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 158, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.499 [Thread-4] AbstractProducerCreditsImpl.checkCredits:184 [] DEBUG - not asking for -1 credits on activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,BYjREAStKM-0e-e5OOY86CxL1g9WrAP01m5ROYtsGCMHTh3wox28vU8_r7Y9e7QwRiDtJ065Xst_zM3u0oFx_K7Fr-ZkN1GBxPR6aWEKrm4nQA,,&typo=1&ancr_add=1>
2024-01-10T19:52:34.499 [Thread-4] AbstractProducerCreditsImpl.afterAcquired:98 [] DEBUG - AfterAcquire 333 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,r4o5lwz6oA1VbUpicHhTmEpC9U7BwWBMw8ZYMcrl4g-jVFykL22Cb2WJIZLOqv-hLbxRqgThYlmO40Jo84JsPHe-IYBjUU8WMQ-jy1lZqxq3YQk,&typo=1&ancr_add=1>, pendingCredits=65203
2024-01-10T19:52:34.506 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 106, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.680 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 155, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.683 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 21, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.684 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 129, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.686 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 57, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.687 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 174, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.688 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 176, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.689 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 56, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.690 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 108, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.691 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 156, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 233, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 112, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.693 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 187, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.694 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 110, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.695 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 144, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.696 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 235, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.697 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 28, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 47, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 30, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.699 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 77, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.701 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 65, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.710 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 70, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 34, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 159, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.713 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 76, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.714 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 119, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.716 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 59, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.719 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 87, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.720 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 80, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 166, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 149, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.722 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 137, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.723 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 163, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.724 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 179, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.726 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 181, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 17, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 58, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.728 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 32, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 49, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 234, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.730 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 143, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.731 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 42, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 136, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 178, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.733 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 55, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.734 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 95, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.737 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 48, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 16, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 146, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.739 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 22, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.742 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 43, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.743 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 130, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.744 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 36, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.745 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 60, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.746 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 117, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.751 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 14, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.752 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 81, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.754 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 93, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 138, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 100, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.756 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 62, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.757 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 125, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.758 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 15, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 99, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 44, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.760 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 172, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.761 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 84, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 79, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 122, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.764 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 64, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.765 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 88, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.767 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 151, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.772 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.773 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 96, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.774 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 27, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.775 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 73, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.776 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 90, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.777 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 134, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.778 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 35, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.780 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 118, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 186, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 41, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.782 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 31, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 150, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 145, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.787 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 127, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.789 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 132, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 170, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 142, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.791 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 53, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.792 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 86, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.793 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 147, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 115, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 12, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.795 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 135, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.796 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 68, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.797 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 103, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.798 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 141, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 165, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 18, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.800 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 33, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.801 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 116, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.802 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 124, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 26, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 50, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.804 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 75, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 85, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 69, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 97, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 101, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.808 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 153, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.809 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 107, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.810 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 175, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.811 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 66, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.812 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 111, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 152, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 45, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.814 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 61, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.815 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 113, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.816 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 105, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 162, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 171, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.818 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 173, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 184, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.820 [Thread-15 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.afterAcquired:98 [aoy5vnfnjetz,6ek4wlg8c0ad] DEBUG - AfterAcquire 781 credits on address rpdm.v1.authentication, pendingCredits=63974
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.user.group.query
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.user.group.query, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.user.group.query
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.canvas.state.query
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.canvas.state.query, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.canvas.state.query
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:184 [z7wx0g93sdr7] DEBUG - not asking for -1 credits on acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.afterAcquired:98 [z7wx0g93sdr7] DEBUG - AfterAcquire 505 credits on address acf62840-864a-4eb4-a902-4861db94eb93, pendingCredits=65031
2024-01-10T19:52:34.824 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d43c72b6, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,dWlUyTFJQxRIqOG8poa5pxge5mYVJ48dUXfACv6LUtKsZzFSVGKyZagOScZPGCnB6cxsyzdobSyk8B6Bwd79504jcLncGDwqMeY-9K6dpOs,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ZNE9b4nkKfo7M-_v-eXIZ86i5mBDuVpXhcuhH7d4Z3TXAYIWBA7iz2F7WJKgkGmWz9vMjcTKd3WNKgAqq_OrNuwfMP9rqwnZZdfDvOywCznI2b71&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,ud4sg1bGnGSsxR72jGNWKmRRRqC9RhLjc5Fa_mAH3joB4qQCFRRYETYRnfp_427BSgD3-id4mG56z7G-3qi9hl8xOglhqXamz4f_kpydTW4Drw,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,5ea91cHeAd9MIKFvy9sbzhVk4n_LaL3aA10YRDmCls04v12YLb2XMmXw6eBUcLVuIE_Z644AmEt_e0ljvQnrS_I6HavZOLh4zjfiHWUvesg,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,jDru8ObAZFnt6UJuRwWa5fbG9hi1-FGT2jjgjYcb0j9MnKP4NE8WTbKBHSHyKvKnv60VBWqMlES8saZuop8zduWr_U-bg4uKdFl24xth0Wc08-CvUBpd&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,FCoOt1eEzJLmEhVMnIIHAKa-kTFceMfQD6Kcwc4ZSxQI7u6ROHuAf16piW1PXfz5fwuSX1g87RD5aUJh-U4htAPPBkPWT1OqrJG8TvqS_LoMCTdPQPVU&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,RozNWdaPDnLkxzHk0PIIu7yOI0mq7J36SJtiYqHJSj83AaF6otFCOMT6S0SrKP0i5_HdKBbJFOfyhgDON-XzQcdSOvC2Z4N90Agiv9ZXimQSY87tDA,,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3xUOFzb6MCi6u1j2LRQL6CE5BWbu5eM6Ir7EsfF2oCaYUBtqnA1r3VDYEmgBiVBb7QypDxbOzY9knH1MLZZ3oikFpY_p69_553LHOTbV7g,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,9xoBaA3kGi2K8zQ8BuNLDq0dYmsnE1nuJw9sP_cDKKpUiUl1eHBvRBJF127g2yN9LNqLqZkyPVSq8YusAWP6k6MCI4_4n846oL1LY84Tow2q00oOukNL4Q,,&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,OqfGI8b5WUrx4fzs6Vhc7lLcb55MI-F2H4j9ic-9Fbs1_FhBfTLzdobmjbVxUHuNV9kcphYOQdlHoTf8bN7hu4yIeKsdhwLW0rasAMuEm4mWhNV8t2c,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,ml--e7XB5_TgtwWFh6lw23pXHYNiKpjFGKoUKmwrOwHyS3lnFQ78O6WUjq-OM1ITxXTrgD6P5K2E89TnqOgGcaOyn6aLaVIcvPhARMyQEqlFDI2mcnYwIfwADKB-&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,olHwwJjC2TUJmGxt3e1EpgRMfVMAAdmudKrqZ5ZByilJc51VAK_ZZHGL1mLqUqv59mtQqrIVOT0FHxcuVGnfmTsHrtP_Lfh6vEjgZmLr0Aa8EqAKSU0BDQ,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,3crdMkhGhlkrJXp-HsjZIEJOjIoajfvvI_g-uTOHgEc26-rm5spEwaDR4tTrIOKeN5HSQz3HfKcnanjUMfuquaMfvk7M7K2AtlUVxZdR9Q,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,Qv7RvLuSjcsV2_72qHHLM6lmOI05RppEGGyR8280SVuKWQgOAl-EeFo8ZDeB_9Hw2f5ZgHbKDJbuB11rT-FBX6wf71ASJsIGNZYXzHn1HnNPy3pXaBaEgvrJvA,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,7hPEKeOrrW0p5qUGMytIu7avFNFhxNhXFzKgo6a-hDqm4ky01dphgxAxTpF5DmP3oUrVV1uWYwMbJuUFv1IoX_o9-H7Y6xXWdj067c1A&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,dU7lk32qJcXeW-cUtM1E65BB47JiLhSgJYTqhLIX373Pj0miueYi3kN7MYtAU5zT39qOtzzAiHcBVYWsGDB3N8R31seI_xbm-rQzfd0jzb6V1rSLy1kotiI,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,BtR0iFYtXaxmAlygBDvC74BLAVELzJivsFTs_YS41zS_XagPPAr0UUKW1lNrafWE3WxaZcW_S5WbNn583_uHQ7aLAA9vskUJfxaMwUrM_DxAOd2N&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,nW_o_ayhGlEOytZThvhNsFjqKDq7oEaYAEjw_jVKEHAnXglYBlSGSA3yj7wyG-2DXRXiRZSD11UwsKx7fSGPuHkUUSoULIYqacVjSrfmUuEfG_A,&typo=1&ancr_add=1>)
2024-01-10T19:52:34.824 [Thread-19] JmsStaticConnectionPool.onException:78 [] ERROR - Receive error occurred. javax.jms.JMSException: ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:714<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a714&c=E,1,3FhMyZ8H2FihvFzilmw5MqmH_tkqp5vHbCOJsN5x6sQNSLyFELuJt_Ej2M_24CH42Xx0iQesALGDJK8HHdR_PDaLploHa8R84-uwfftA3DUM&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:735<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a735&c=E,1,JvpZWdXDc673FDsIBWz8WgGu29y6j8H0zAczvf_UXYyUAzEMKtvXJ3tMkleaFsrHU-QdlAUi0bWddjnikuurWEEMfZs0jIDWZ3jnrx2__Vbg93yZ8oo,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:868<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a868&c=E,1,AznnL1cSXDTbCj6iQ4ovEeP71-oopbCcH0laxV9dB-Y_kBuerNfeOAXRkRLybAqgqQBjJZKBNFEoveCJjTGncMCguHnY0hvsdZuKIHz6BEhvRX7N&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:856<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a856&c=E,1,vBewrTh4rfp_ezwl572jqeiy0VBShHwQJNRT99yv4B54VM5Qaf2hTHrOq33emXwk_WSFrcueJSCKI90AxcN4jUznfP1ZLzL1dJdUz01zK4vN&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:802<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a802&c=E,1,EY-ry2DzdJloXWtaemW-dWnRy1-rr2mQFFjoCbmWjX6xHCNZF95ldY2zUgfhOhSQFfzm7eYV0PdRUTOwJrx9jCD4U-B5ztFqWmvqEmJQuqdCyQgypA,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,gHouFFMo7plejV1zi4ureD9QEDkW-WjvytRlH6DOyEt7oMO8ULkj6APov-gDhxtXAspTlk7qF6RYw-RrCqfSrXlcE0Cm45J61qJHt5Yc3zuBJJyEX8A,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,17MGSyeFWTwZKhb2DCqWeyFvdlinlKAunc4JSHJltjjiaCeT_1UjXFsTMtEiF_NYIOMI3brJbPkRW1QkJIhe_Br44jy1od8-BI3l37IfFJzZbcculrc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,F6NyfiAiJI9fxHnOQnnpO7dLyoFavqwnNivvLL6x2LO54G8SpNhXjlUN748SkNWwe_oCD9JhI7u_BOk1uNbwSTYV0lXRyewBrwvvysSEOSCggiFCFRgjtA,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,TnFdrKl6nCZv9PR6lSsRZ2YN0r8LqpGQMRo3oHlgyE7_AgUMzrW12R4HeB5JhX8Akv-1iKeVGrnTLV2WeAbBOCM-fukhfRPsdUgq5pwe0yU,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,vAtPtyOuzaufOELKjMWnOBMrJEOShH0EzH9G3WL8gak4hWHmnVWPL1W7c_YnxRH2IerBQErg9VJApOiiX6TR6tcPau3xqRHrPyrsH70KHg2v&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,pak9EX0xrMkRTN71voUljZIOmUvksqdCCqWdwh8fq2R4OZXUQFIHNH3P9j0t-wpQStfSs_ZsT0a9BZYopaoyMSnMkN3xqvXn8A9dTehv&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,acJEflBFObRkFNBskLYk7AARfS1_CzZH3yFGYB9r_Ype7n0DAWduZhGpmImVc9k1pTnmFoA8L8UukMVvWBinxujU6r75ipmzh4I_cNQ5Gzmd97g5SAU4dRsf3Hw,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,aM5tVNMvOtLUxuoV8Dez1Y7fAS8KwME3peX3IqP_QPViIWgkZQahJW_1BMBQUo35u5h9hGJ3XSyDgkECwWOGDn6a9W-gctjh8WvUKx9yQ0sSLwheMcs5mhR2yVM,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,fUWcgLX3hupjGCjPYEByZ8xfzFx4iN-VnkLB9MarNu6OEOfrwnCkvJvtu6e-ZV-FREp0Ssisi_fezDu18eP9adS4mmcPPlJH69vnf0YDVg,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,KuAjbB9H9ddyi6unfXrVdNXylpB7l6qLIUfJARScyUnc7iH1GtvO0s1MAtB9fwkdqXq8idOJYZs_rg0iU-sN4aV24ID0Os5tsyvB34ba3ve2D3diROqixzkQuw4,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,EcYd8WT9GtMN9IXo58gsQpj5uNwcdCRtDfa0zG9t2jpvfdaSX5r9vLey2icJ-4wo3EyBMyUQzwyqYwC5BMfe4yHGjyo7h8jxmjcEKig4V0KyVa_SVq58NBT8&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,_a7OW7NorEoCm9eOnYjCi6TFl2__or_ickK-DwP4rrMmRX2Rj_TKaxpGjSAwCJZasllt1gefhShoob68MthMyp6bxv9N5fucG2HiF0g3yK4kqHTZjiFn7kxZ&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,-ttWHQzQDZ94E-HqKtYTKV2nuApkTk60Zy25iWFIykqfNe0o2seEul4UE_gETZr08B40bJjmZKMHq_-R0M8bQ8Fbu8twSkWx9MPLFUhc&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,624T4PuCyLmsS-UG0IoXRx3MMz4DqnU_1BuWfOz973WSpqfLsFFDAICu75KCoCwAs9O8WgKTEcBVqRhhf1JUqI3Znxem3LsG7Ml0Td3DnJJUoGYR0BE,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7UL50c_ahB0IhSHbe9OlGABBKKE4OFyNM67SjmNrSo_LQhfViHAlLgvwdlRpbmDdymI5VaeOEklHjvlweLBz8Ske6wjc8J3CzM05VYenh6DzNiI,&typo=1&ancr_add=1>)
Caused by: ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
... 7 more
2024-01-10T19:52:34.828 [Thread-17 (ActiveMQ-client-global-threads)] ClientSessionImpl.acknowledge:1225 [] DEBUG - client ack messageID = 10054445
2024-01-10T19:52:34.862 [Thread-24 (ActiveMQ-client-global-threads)] JmsServerBase.logRequest:117 [] INFO - REQUEST IN:
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,01YVdf0IHIl4w2xIwUh5i7ToRiQfv6TxT-FRWSNW0n6e2aucHkWVyKxkYJTpG1jt7_EUCuQVS0NWGalDyf11AUIb3qVPj67IUtjJB484IdLKrHgbmSd-Q8Vg7AKZ&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Tuesday, January 9, 2024 1:57 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Both the "replicated-failback-static" and "replicated-failback" examples run without issue out of the box on my machine so I'm not sure what's going on with that. One thing that's odd is that out-of-the-box the line 122 [1] doesn't match the method call in your NPE. Perhaps you modified the example?
Aside from that there were a handful of questions in my previous message you didn't address. I think the questions about thread dumps are the most pressing given that you indicated you're deleting the pod. My guess is that the client is getting hung in a network call that doesn't want to return due to the pod disappearing. A series of thread dumps during the minute delay would confirm or deny that.
Justin
[1] https://github.com/apache/activemq-artemis/blob/2.31.2/examples/features/ha/replicated-failback-static/src/main/java/org/apache/activemq/artemis/jms/example/ReplicatedFailbackStaticExample.java#L122
On Tue, Jan 9, 2024 at 12:47 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hey Justing, thanks for the help.
We are using version 2.31.2, which is pretty recent, if not the latest.
With regards to the log traces, other AMQ threads show similar behavior, and our system isn’t doing much until AMQ-client stabilizes.
I am running on a k3d cluster and the failover is triggered by me deleting the dm-activemq-live pod, which k8s then spins back up.
Also, our slave config does NOT specify “allow-failback=true”, so the spin up of the new dm-activemq-live instance does not trigger a fail-back, which is intentional to eliminate failover-failback chaos in such a short time.
I am trying to setup a simpler test case by using one of the examples, artemis/2.31.2/examples/features/ha/replicated-failback-static, however, I am getting this NPE on line 112:
Caused by: java.lang.NullPointerException: Cannot invoke "javax.jms.TextMessage.getText()" because "message0" is null
The NPE seems to indicate that, you cannot call consumer.receive() again even if NOT acknowledged, which seems to contradict the comments on the code. I have run into this same behavior in another example as well, artemis/2.31.2/examples/features/ha/replicated-failback.
Can you please confirm the expected behavior of the example code in the 2.31.2 version? I can then modify the code accordingly. Or please point me to a working sample code.
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,lIOPl76F2R4EtenIAEJMeQ17kfmHQ6buC7G1W5TRefOcSfxvj_-3dLXV3MKDkCTkac_65Rw0SYmof66i91zGwgb_SS1nKaBpao_Lw7JNNk1mkE33&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Monday, January 8, 2024 9:20 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> Now, the backup process does start processing msgs right away...
What evidence do you have of that?
> ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable.
The log snippets you provided show a reconnection attempt back to dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to dm-activemq-backup-svc:61617 both of which fail almost immediately. Then right after that another connection attempt is made to dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e. from 20:03:44.874 to 20:04:44.933). Are there any other logs during this minute? Did you happen to get any thread dumps during this time? If so, what did they show? If not, could you? What was happening with dm-activemq-live-svc during this time? Perhaps there was something weird going on with the network related to whatever you were doing with that machine to trigger failover.
You might consider simplifying your test-case and then adding complexity bit by bit to see what triggers the difference in behavior. For example, run your test with the client and both brokers on the *same machine*. If that works then move just the client to a remote machine. If that works then test the client just reconnecting to the primary broker upon restart (i.e. no failover). Then finally put all three entities on separate machines and test failover. Hopefully such an exercise will give you a better understanding of where the problem might lie.
Lastly, please indicate what version of ActiveMQ Artemis you're using.
Justin
On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
I’m working with John Lilley on getting an ActiveMQ HA configuration operational. Our current HA config is replication with master and slave instances and with static connections, and it seems to be working, except that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect itself after a failover from the live to the backup instance. Now, the backup process does start processing msgs right away, but the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable. I have included the relevant log traces from the Thread-0 ActiveMQ-client showing the Connection failure detection all the way through when the client is usable again, over 1 minute later. I am hoping that, based on the trace, you can find something that, perhaps, we should be doing differently in the client side?
Here is our Client connection url:
(tcp://dm-activemq-live-svc: 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
Should we also be including the corresponding connector names?
In an earlier email with John Lilley you recommended 2 brokers using “shared storage” instead of “replication”, which I will try next, however, since this issue seems to be on the client-side, I want to make sure our client-side is setup correctly.
Thanks in advance for your help.
Lino
Thread-0 log traces:
2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
---
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616>
2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617<http://10.43.11.76:61617>
2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
---
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616>
2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = ced14fbc
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617<http://10.43.11.76:61617>
2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 10, performing as a failover operation now and recreating objects
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,hqiYs6vnJzfq-X42G9W9ohJtYgn9LRPR-82RHxMyPnKUD5xBwCbo8W6b-PegfmAGwvEENlDO0k2IkoGFFegGxR-H6J72UHHlpEFbGtJzhCF3sucQ&typo=1&ancr_add=1>
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,KX9atZvl9XxpGPWobW9hXsmaLrVekfw2221zvD0PJ9Ik3fEQVyHKcZZCe7MQRciamxD-T-WDEk5Xa7_cAt0PSHwVW3jG7zzhq8XH9XIRC8EXmg,,&typo=1&ancr_add=1>, needed = 65536, arriving = 0
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,ZOCwWoQv34cYLuw9hZvJzBHfSrX6oQeQ_3-SetFMJa1hRC6lXgiKsAElOhfZ_vfnBhRGRJPZjSOqHIbwWO2ltSYYY7ElQjyItp0uDo79_jaDkw,,&typo=1&ancr_add=1>
2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,obbNTaFqn6mglgyZXy1S-0nBLNVtEYlkZBwAEKSGDGN0RuqlR_-7fIQPW4ELNqVFJdenAcKh65C3yQ2b_Lef7W01Tst6ZTrmI66KfHLae572jXv-HF3O&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ytVVTXzfq-J1NXl2ct7T_47F5EbfZNQTXP8dlqw720EiCe32c6CLwlpk0-v4MzUD61FErq0nkRTalFpcTtIN8TDeBvz70rzb__TyeIDm48ZKfFU28qQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,KQYUuEgOaR6V8a_ZC_xBpKZmDneB4LTL1cQzn2Gqtla8ntHJ1fgRs9KelLgvbMd9zvBTg8maXUerYZYQZa-DlfK7fon0JZ0O74PNJ1SOCHqIPN4fZQ6Xv731OGOD&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,OSVh6NSORxz7qZ6eIuUDgWskoYfyXT4NNdjJYGMbH7HeWEUiymys2DnptjYKdmQfqjfHmI-egRZ31yWwd9MKzEd-gbbsuTUsqn_46niKaw,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,DM2boEBGg44zzcd0m37ITkOnfDrkqLE8ZSFGJpEI4AGo2GyOI2yKX08AEkKHxw_UnDMF1vccYEvESFDXjXBBjjIGO5TEsMLWZHbJF3AOIEQ73uc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,wipgGeA7UVZIy1uuqvuxJEx4BA8TvYfSYmO617IgrvnOo-z7sn1vUjXFqIEU9M1HmedBVoHxB_LpvuEZ9p_biioPM8V-V9eS8i8oST46GvBhZZIc&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,0FNTjscT_d2CNIid0J8d_x7C7Peb6_neE-a_MZeKqkM1o_sPGujvpkXuiLv6kms9wSFyYqSux86C1MX2R8WKM8IZBsjKt7sCs03U2lRYlVbw2eTDg0ihSEV1nA,,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3yKPqxpIi9n43QJSCQiysKGlEYttZ5Dmuirc7jUAWNz6orLfIUCYwdDEJ3eU0QiHywsNAJdWqrvnx5rZK6-8TTJb6o8tr1WSvSL7omLF&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,6B4qLrj7q9DzDb0Trp9xgst7cOBhbD4hcEcGyPFul5E_mh3_oGmColB9YQkQTB1gJ6x5Mcm0VwvBLddpDpfFkcZwlzb4YNR9Ym5fcDclFW97pIrQ9oG4&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KqR93A_gEw1FpU7__PsYjSkOC5LH0zFTSKu5K9YlCdVAz3h_c9EkBndO7aliE4RSbK_qslEZp0b-YzJfCOxmB22jAfplsYxguMl1qjC0rWpNjfr9_fh4RjNL3AOK&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,mhChlKkihYjDw_Kk5SL1d_4eb7zhBlSzri_nwjOUsrs6MAvEfDAAjia07yTMUU6CQqc0WbDS2Fn8HJJI0UwJh1N7SKiz9dsP1gJLmgY66huLhXZE9c-3ECGXyIWQ&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,nn8IDmg6XQJLLXzH7kunXDNZa_3A7fUlnHdRCLoha7gQWJO1QeY32duGFAQMYaFHsfIbHbv142mei8HfCqRwrpuUfikYxLkFoXaD3pORZb9roBJs0H8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,DPLEvt-MPmmzv78LEkeXtYHM2q37m12TxX-nqjuIJurQdeXydsSakco9t_O7jgBbQ6y115xxdDb8IonFZbM86zgRDLcvUmeOSJuDmZybD_6JaTX0zpbQ&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,1vNFV-mDaLK92lJo0Mq6jvkCDzuwrSHT4lp1nB6KTzOCSO6comH5xNpEcvtZ_9pc0cQCD2kgDfB5xdxg0tAX2dF5oG5DHY6kxWXOBLnuAwiz1BkEFhw,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,aLydz0SGsclotqCro2upcTg44B_Ij7ckjApLu4B5JUwZKKdMo5z3RyUR4ikHFY8L3t_yB87Zp5atGRwoAPA1L9Mc9c681UjFrG16n-udVjeHtxQNxx8V-Nug&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1mZX-u-XHEZ_e4Ut7YETRvWm7U_qkC-q9QaPeHah4K4yNl5foyM1VSExCtYiisknkelDj6_K2MUHGc65Kd-6-JG_TXLby0OPWodJVjX3BtKaDNBUzusgpzT6ajL7&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-nbEPDxI7J-DFvbJfVVt2Cs1Vi_hN5bw-8vQF3SlM1nthw3VbJLWmS1h73cupUxqBoHCImTzDqI1m5gutx3ZriLpLpU7IqutHCqZuT6CTVWACPtj&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,-lUcdhDzaHWdAWLCc0IxWCFkKtKHxoEd64GWhzXAbeuG_Zm9mvn624BY6uS8MgQn-s2TMF8PbEPYVbOOy6BJuI8ByeiuAF9Y8f2I5wib&typo=1&ancr_add=1>)
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,T2UV-1dji1RODRcU4Ff4j_IzfTH2n47BatzyLZcEOU8VROFr3GS8re7CjDcnnHCSaPEtR-35J3z30dnYcz5K66rVsMVxgDiyh0FvEDKnN67naHjYbn0NpzAc&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
Posted by Justin Bertram <jb...@apache.org>.
> ...I’ll have to instrument the code for that, right? Or am I missing an
easier way to generate those on-demand?
I'm not clear on why you'd need to instrument the code to get thread dumps.
You can generate thread dumps through a number of different commands, e.g.:
- jstack
- jcmd
- kill -3
You can execute these administratively or just set up a script that
generates them every so often.
Justin
On Wed, Jan 10, 2024 at 7:43 PM Lino Pereira
<li...@redpointglobal.com.invalid> wrote:
> Hey Justin,
>
> I did finally get a chance to run everything locally, and it all works
> smoothly with failovers in both directions and the AMQ-client recovers
> within 2-3 seconds and with a lot less log noise.
>
>
>
> I did not get the thread dumps yet because I’ll have to instrument the
> code for that, right? Or am I missing an easier way to generate those
> on-demand? This is happening in a k8s pod, so not easy to connect with
> jconsole.
>
>
>
> Thanks,
>
> Lino
>
>
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> Lino Pereira
>
> C++ Developer, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Wednesday, January 10, 2024 3:02 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > So our issue seems to be only for failover from live-to-backup,
> consistently.
>
>
>
> As far as I know the client doesn't make any distinction between these two
> cases so I'm guessing there's some kind of environmental difference that's
> making an impact.
>
>
>
> > Is there something that we are perhaps not doing on the client?
>
>
>
> I don't think so. The connection URL you provided previously looks good.
>
>
>
> > I will extend one of the example codes to use threads and sone of the
> AMQ-client wrapper code in an effort to isolate this behavior on a simpler
> code base, as you recommented.
>
>
>
> Before going down that road it might be better to inspect the ostensibly
> low-hanging fruit first. Did you happen to test everything locally? If so,
> what happened? Did you acquire any thread dumps during the delay as
> previously requested?
>
>
>
>
>
> Justin
>
>
>
> On Wed, Jan 10, 2024 at 3:04 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hey Justin,
>
>
>
> After starting fresh with version 2.31.2, the examples work out of the box
> for me as well.
>
> And, after ensuring that version 2.31.2 is running everywhere, client was
> still using 2.30.0, and refactoring to using the recommended shared-store
> HA, our AMQ-client is still behaving as before. That is, failover from
> live-to-backup, client recovery takes over 1 minute, and logs report what I
> posted earlier. However, failover from backup-to-live, client recovers
> within 5 seconds, which is the expected behavior, and our system is happy.
> Please see new log excerpts with more logging included for the 2 relevant
> threads.
>
>
>
> So our issue seems to be only for failover from live-to-backup,
> consistently. Is there something that we are perhaps not doing on the
> client?
>
>
>
> I will extend one of the example codes to use threads and sone of the
> AMQ-client wrapper code in an effort to isolate this behavior on a simpler
> code base, as you recommented.
>
>
>
> In the meantime, I thought I’d ping you in case you can think of something
> obvious that we might have missed, especially since we just moved from
> 2.30.0.
>
>
>
> Thanks again,
>
> Lino
>
>
>
> Our versions:
>
> ARTEMIS_VERSION=2.31.2
>
> JMX_VERSION=0.18.0
>
>
>
> Logs from failover from backup-to-live:
>
> 2024-01-10T19:52:29.588 [Thread-24 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-backup-svc/10.43.229.161:61617 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d7ff7885, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,O0LJ4VfRCemWEaMLULajHsgUo6YseAVDyBlrTm2G50vM20WhlTUV0ueY1DxMRAMStXAoc6wy5Xx3x1vi2XsnE5F7_iNbL-MZ03M4IS9G&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,VdcejKqF9B67eWPDv-PP_0gR6PEmUxFYHHtW8kyn3Gzg3q-HHzL-Lr7oztjj2LQumPJWO-1KF4OPavU8c3DW4zPgrUsidhnnCpoMUUn_&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,xKCDuAN84vJBjsCp75epCz7V1lR05TFhETdvpPcKntBTPnQk0zuuQSei7-ly1RPUUsHDP8ZAe-IevqwGtTGfEB_yNzw90QcOpEx5BaXOW6E,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,pq7BenEb7b9i-FYz1LnTy-gHYDdV-28FiSv3pkV_i-1TQZtRxakbT-rOKkgLgz2IDx875Dk5EE_OrwsUcew9TIsvrcZZZ9ZIlwboxMEq9xnR5odpQL8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,8mmrZ5alRiVlyEshqjlGpedpt7TuUVBJOy8DDm2qHLGoY6v3pR76llV59sMsYwy2v4rwLqiD9Rcf3Hg4QSCZo7rdO1rQgMNapCKji2jg7T4IJOk3OAeLnIs,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,RLTYzCoyknTJ1GB9szMq3Jg02QAnt7juedv1HCTYmmA4JU1DO6MlJhkvzOTBqHbeeOdKSvwkualBrhGt0fUk7Nl5YLzlgfepqTWcDiBmXQ-_xSMcAHPw&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,iYtEGEWoGf58Vq3CGAxBrH3iWZ1jWm1N1b3AOFPNlx0kCahcxHEwXT7ZAQ_h3ynXTeKJSkvmBiGUkbqE3bH7tGw-Gn8VV7R7LaCkxl9Dch5I4kQDfJDKSg,,&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,kH54RADAl8RcHDeCxJW9b-P4YJJFElj_Nbh6mHtmwlFdP2tKzzbK3wkAXg9tVqhIlsbd4gr2LJkn7DWZ4I-J8WVE-7UovVbVakuQTTxc2pbR9vuIDiEFztc,&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,V9-XUYgB7GoqPyGhEdE_CmtNood-yvXw6P9wRIqQOSYSWuTfiF1bhonQejBQfgJdHq8k2TsE1RAvcSOp_WzBE-ybl0l6rc5pIuMGdC1LUITpdk_Iuha7urSje4Y,&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,RNgnFH7dBkD27TX353Mc-kqneOLcqFXEgpGywxg8VduTTYNyYFt5VqkKfirozQw9V88WOI8XK4v9dvrtPIl-3u7cp673GvKxq3L-XgqypuDtSnkRIYnh813U&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-backup-svc/10.43.229.161:61617 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d43c72b6, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,L97BTn6AFLbUpYgHzAlQm9pDJ4Sn--TlGDpz05xOE3xd-qn9krfLJ6JzGTaiSpQwuVjuQ8KUPfUSW6N18pgw44-ay7eqYb2h0p3sxehZeXAC&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,PiedcDlDv-nxD-EF13k9j-sagD4clTOVvCs6xyDQvnrPzDUxc4xAbnHg0d7hnqeG4cVJZaO42LJvoNm59E3SmClgLAft-3N7F898HO6Obbkm&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,G5SZ7tGYpGBQzNRxdmIzsp4hWlUsmbY3_881k0N5lFoFBwuHt1dMCkKHI8wiZRGpS6qDY_UDBvDmKVQNRIHXIIyg5Df2nnWDCTLnbvWCO20fW0PC&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,MsagygznR7ie9qgJLlMogXq8ex-X4ST-fg_vbVbgcBanXur80DVykS1ftnzq0e4bWLE0k8EiNmlhlezemqTlBEncWMJz-AQAKdTxEk4WyVY,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,n7l9CUpZZbFH3ca7Iu2wxM3kNfged80mMRcu0Vn9wIoZz7aKzCP3c1OVR73iqF0oLbir7CjMHKAu3sfrR2cLs-EP7orqDWQKQApkzTZra9CJag,,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,PePvk640nUsvwJWNJY39J2R954rSgiFVPPSQLOsjYskMxtDnDVBuzQiZ8ut5SAPoqzV9uYW6GX3ekOAAoou9eu6VpHa_oFc2-7d27W00VBLU_o_Zv4wEUw,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,REmfZe55ytZRsqZvFkCNsYUcRO5Yd8PZwIlQfGystV0qUeF2mXSYaWKgleI098Ac40PND4ilF-ukqEWDY8N04AIRQgGgKGpG036xx7jNSc78YS6UbKWiMNUHfNc,&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,nnqfN76WKNs0ZYhVPzea17QO5aWkwmLqbfYrmQJ_qGQIgTzCL4PSXhjU9jy4mhbCUfl53cwF4ISYabr_zD1GUvCgv5m4dYW1_OROCBpe&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,vlJ5vYZAfrlVp9kL50a_jwWhqVW8-LMUgzeBb6mn-udcS_fMR1yld889bXgOXTMDiutFqKYX9XPh0RtxeXRLtnN31WqeOlH_CZv4_ZyhnOwXvm4J&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7Ba-VAkMm3x0E0f6F1f73TH_y9Ylo03hx9DEz4kqWzvrXy0gAv3D9zIM35eXIaB95UFxo72DkMpb0nKW_a-BIF89JobbRqjyVNLdC5K3L1EM520B&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:29.592 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:29.594 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:29.593 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:29.597 [Thread-6 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d4cf19d5
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.600 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 7b2392ac
>
> 2024-01-10T19:52:29.602 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.602 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.603 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = c1fb4c82
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.610 [Thread-8 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 06b7b3a3
>
> 2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:29.611 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 65680be4
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.624 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 97a109b7
>
> 2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.625 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.629 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 77b2de2f
>
> 2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
>
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:30.612 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = f92d7cd6
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.613 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 1d44895d
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.614 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = e8602acd
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:30.631 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 0a217bfd
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.632 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d2aa00d7
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.633 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 9a4f639e
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.634 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = ecdb1cbf
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:31.000 [Schedule-Runner]
> SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service
> as system
>
> 2024-01-10T19:52:32.001 [Schedule-Runner]
> SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service
> as system
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 2/200
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:32.116 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = a6b3d488
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.117 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = c0627cd9
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.118 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 71523eb3
>
> 2024-01-10T19:52:32.119 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.120 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:32.134 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 2/200
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:32.135 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 68efd0dc
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.137 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = daddef73
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.137 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 84813ce3
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.138 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 533990e9
>
> 2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
>
>
> 2024-01-10T19:52:34.370 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 3/200
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:34.371 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 1d8df71b
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:34.373 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = de20e193
>
> 2024-01-10T19:52:34.373 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 3/200
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:34.390 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:34.390 [Thread-10 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 4661fd5d
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:34.392 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d1c14a74
>
> 2024-01-10T19:52:34.392 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.404 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-10T19:52:34.405 [Thread-9 (ActiveMQ-client-netty-threads)]
> ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up
> backup config = TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc for live =
> TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.405 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 38, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.409 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-10T19:52:34.410 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 37, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.411 [Thread-24 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 10, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.411 [Thread-11 (ActiveMQ-client-netty-threads)]
> ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up
> backup config = TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc for live =
> TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.413 [Thread-24 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,S5999ok8Om6JjNfBLhC0R2hYdl3dGqPWsLpis0MTx38AhKXPNqMMlH41RrDz0UZsxZzSZbLvUEOYFEa7hwM4N7gAZfCS9udcqSmOwhLg&typo=1&ancr_add=1>
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,CF5ZPcR0aBH6uaRjLjSeWoBKLnRwUOOgvrD5MzWuerKGIs1FEa5vZF8rdF1kWSKbLwGQlm1IsOeyey0f1t7eiA1ApUgs1EDhL7ilSV0c6rFxEanucXI,&typo=1&ancr_add=1>,
> needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,nl9VlpvL5Lx7cxWUON3eMxxXV1my7fPq5WQLaWD1ozszhEPr1wVfqCwFilScj02MzRACloIhLZ8CgyMWoa_OqCu-Tf4folOdvKvtooRq&typo=1&ancr_add=1>
>
> 2024-01-10T19:52:34.415 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d7ff7885, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> ClientSessionFactoryImpl.java:410
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,awSP7U_0_MIm7Km9lwe59SrlpKsd9iUMA2y266CCeNZS6GJuEExdDYVS6EVMnlmPlVN0W1aqSx2Rss2Dm_YIM71s5j7OAUz6pAdGaTqnU_h2RyaoD6KyuoU,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> NettyConnector.java:835
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,8gAKfG0MZn0OMdhCHJ98evOezuGoCS0mNbGWplSwawxhF0ed3tPq9vKWoWUbn2xmoLJ34skkCTpI2tUuevSCeVNT00tNpFlOJlyAjGWu&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:666
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,tXm2hjddgUkJac0QpMg4nVt2gwg5e3ORdql8K6xsJlBZIxph-NuPt1Ks2iB0ZarzoFELPAuAerVMiQpEU3nkq1JiqLAju26hE6Y2c3MwmKRTj66O8tsfQ-WFSh4d&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,P0xsXGcezpL-KzlLSOt_rfr6b6RF51B2hthJzXMGD04QeyLyg6AnMvF4M8ZOEwF90ojRdT3CtOy0x9HBSpGK_mmF_6Zl7Muz2TKrZRLAxQRxhV5XnvYlwQ,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,qctWvKkM1LAyegiROEl3hsiEN3dPcBcH4jlkBConOP-G2cgGl6X6XG-kdeQuBBOC8BmtTf-VmlbpJTjC_wznn8I6g7pLOg1uof2v0j_r3yyK7U8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,KRtNFryEe6jhqdT3of2FZC9PF7zGlYTxJ3KPFwO6TViZkyaHiybwHP3JmOhQ4F7yZtorUVRe5phyCswDWLc8lIFBVowIJygjwxLmO-F3xJSonSQ,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,ltH-LxKKSiKev4cs4CgS53B1Sn1z2t9IkxY_vjCKeoXMalJfF0yTujWPogG1a9qkWV5DrAT_qCTk7MdQa3LpkAae5yHTe_19fREF16p6vMMJqGXqLjw,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,66enqLb5-TVz_cB0ZCkDW4SpX4c3v0jslINMsxhHSjXfBWrIFoM8PxfSS7_VWmc_MdfmjPHIKcLm4zfcCe4MggwEWLxB-RjiUMqnmc8ShDJLoit5&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,xM_aKOMjg39ORE7gTvaGKP72ytDNvt6tjanfNTzekduYVGBvn963oijBiSY1JOQeorfgcCf57lVZSYIQ20x8sOFAH5zNexuTI0rBtrkDrtcJpq-S&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,Jr0OyBaxbJX-r7nzGvHtUkNxDkYucNHeuCPW_Mqw5I4IS0dve_XCjayhv7uvFQTdinHTsHqf2ZtZAn_qz1ZnE04PEsE0KWGfHQUj6N5WIKcNhyQ,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,fTod1kMxyNrLe6wmpPVQB1vco-7IrNIs9Dx8h9kWYDCH7xYH8V5kj2iao_X4D62DX5S24Xi08LA-xy2zHwj-LIzSKOA38iBBAlHdMD7qu3lK&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,37VGZcLM1KBT3tttPbLrhK822id0s0HCTimMYb18Jk-8bCbWxvPl2N9_yFFLgXa5pC9DXbKX6c8DMHolUFme8KRWVQdBx_lbRwtpLIOp6O7uiuV0fttkzvI3jxdO&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,zIgfFXrrGxOB_lKTtscVc_jXQlq_pH7TMnaK0IkixM4MVqoSgc-Wp5VV-QGQWmUtW5dggcgtYpRa7OoRHCBvDew7fxYJq28eSZVguCiuBiH9DzdKTHAXC5o,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,b6YutPqRZaL3jtcBxfqj_SiU4M_x_Qx-GeSmBfL7c6CG24IUL7AdjJxpyosXZtCtKco4bSDRxOj7HJn5CWgbgbPzJ0TFuXzJjs7VRUKVhg,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,xdw7r1BqGJuJPUUyo2a9wj_y2KKTC-j8c0DAocpHKzpZZaVOgPcxdng0Xi_JMKvlGYmcUvlTfZMm2FYQ7pTrF20xWUreNaRQqtU6rlWB3kkuPmMNBALyTBs,&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,4iCAaiDF74c7cCKTdUIIlT6Qu6BmbQItK-gaAIrQtpNdie21ccXNS_dCydALepUtQNTgsVsVUs1NavHOugg0IlFK0CjTj8TKnThm55ffUeOx8zHc4oZUYpE_LRsz&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,qBGyQwtcrkAjVKJZ9cZzaW1ivhGQgGfhpcJUS_ww-vfcQVNfD9OjDwDX1XjpzJWYpo08MqSOrKLpgO8d72TBVtSGg2OQ4AmggCv3IUgKeA,,&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,Mm6fpsoFsFLUtanrkJ6cqlk2y1zCJbPDIfqxsd0pS_sfS05oTAHBNIXIUUd4iA8HJY0guikOejZmykugHqKeopxLvJE3UmpX9YX1Y3Gu2ToK-B8aRw,,&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:34.435 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 54, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.465 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 98, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.474 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 20, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.483 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 128, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.495 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 158, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.499 [Thread-4]
> AbstractProducerCreditsImpl.checkCredits:184 [] DEBUG - not asking for -1
> credits on activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,BYjREAStKM-0e-e5OOY86CxL1g9WrAP01m5ROYtsGCMHTh3wox28vU8_r7Y9e7QwRiDtJ065Xst_zM3u0oFx_K7Fr-ZkN1GBxPR6aWEKrm4nQA,,&typo=1&ancr_add=1>
>
> 2024-01-10T19:52:34.499 [Thread-4]
> AbstractProducerCreditsImpl.afterAcquired:98 [] DEBUG - AfterAcquire 333
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,r4o5lwz6oA1VbUpicHhTmEpC9U7BwWBMw8ZYMcrl4g-jVFykL22Cb2WJIZLOqv-hLbxRqgThYlmO40Jo84JsPHe-IYBjUU8WMQ-jy1lZqxq3YQk,&typo=1&ancr_add=1>,
> pendingCredits=65203
>
> 2024-01-10T19:52:34.506 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 106, performing as a failover operation now and recreating
> objects
>
>
>
> 2024-01-10T19:52:34.680 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 155, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.683 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 21, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.684 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 129, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.686 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 57, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.687 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 174, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.688 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 176, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.689 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 56, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.690 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 108, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.691 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 156, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 233, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 112, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.693 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 187, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.694 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 110, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.695 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 144, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.696 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 235, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.697 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 28, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 47, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 30, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.699 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 77, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.701 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 65, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.710 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 70, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 34, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 159, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.713 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 76, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.714 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 119, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.716 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 59, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.719 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 87, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.720 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 80, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 166, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 149, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.722 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 137, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.723 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 163, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.724 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 179, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.726 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 181, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 17, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 58, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.728 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 32, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 49, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 234, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.730 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 143, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.731 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 42, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 136, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 178, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.733 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 55, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.734 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 95, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.737 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 48, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 16, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 146, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.739 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 22, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.742 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 43, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.743 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 130, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.744 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 36, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.745 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 60, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.746 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 117, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.751 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 14, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.752 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 81, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.754 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 93, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 138, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 100, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.756 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 62, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.757 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 125, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.758 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 15, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 99, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 44, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.760 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 172, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.761 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 84, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 79, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 122, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.764 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 64, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.765 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 88, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.767 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 151, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.772 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.773 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 96, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.774 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 27, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.775 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 73, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.776 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 90, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.777 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 134, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.778 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 35, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.780 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 118, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 186, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 41, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.782 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 31, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 150, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 145, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.787 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 127, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.789 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 132, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 170, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 142, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.791 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 53, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.792 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 86, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.793 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 147, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 115, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 12, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.795 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 135, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.796 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 68, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.797 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 103, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.798 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 141, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 165, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 18, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.800 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 33, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.801 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 116, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.802 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 124, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 26, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 50, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.804 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 75, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 85, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 69, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 97, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 101, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.808 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 153, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.809 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 107, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.810 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 175, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.811 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 66, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.812 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 111, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 152, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 45, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.814 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 61, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.815 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 113, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.816 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 105, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 162, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 171, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.818 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 173, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 184, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.820 [Thread-15 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.afterAcquired:98 [aoy5vnfnjetz,6ek4wlg8c0ad]
> DEBUG - AfterAcquire 781 credits on address rpdm.v1.authentication,
> pendingCredits=63974
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.user.group.query
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.user.group.query, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.user.group.query
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.canvas.state.query
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.canvas.state.query, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.canvas.state.query
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:184 [z7wx0g93sdr7] DEBUG - not
> asking for -1 credits on acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.afterAcquired:98 [z7wx0g93sdr7] DEBUG -
> AfterAcquire 505 credits on address acf62840-864a-4eb4-a902-4861db94eb93,
> pendingCredits=65031
>
> 2024-01-10T19:52:34.824 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d43c72b6, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> ClientSessionFactoryImpl.java:410
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,dWlUyTFJQxRIqOG8poa5pxge5mYVJ48dUXfACv6LUtKsZzFSVGKyZagOScZPGCnB6cxsyzdobSyk8B6Bwd79504jcLncGDwqMeY-9K6dpOs,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> NettyConnector.java:835
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ZNE9b4nkKfo7M-_v-eXIZ86i5mBDuVpXhcuhH7d4Z3TXAYIWBA7iz2F7WJKgkGmWz9vMjcTKd3WNKgAqq_OrNuwfMP9rqwnZZdfDvOywCznI2b71&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:666
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,ud4sg1bGnGSsxR72jGNWKmRRRqC9RhLjc5Fa_mAH3joB4qQCFRRYETYRnfp_427BSgD3-id4mG56z7G-3qi9hl8xOglhqXamz4f_kpydTW4Drw,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,5ea91cHeAd9MIKFvy9sbzhVk4n_LaL3aA10YRDmCls04v12YLb2XMmXw6eBUcLVuIE_Z644AmEt_e0ljvQnrS_I6HavZOLh4zjfiHWUvesg,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,jDru8ObAZFnt6UJuRwWa5fbG9hi1-FGT2jjgjYcb0j9MnKP4NE8WTbKBHSHyKvKnv60VBWqMlES8saZuop8zduWr_U-bg4uKdFl24xth0Wc08-CvUBpd&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,FCoOt1eEzJLmEhVMnIIHAKa-kTFceMfQD6Kcwc4ZSxQI7u6ROHuAf16piW1PXfz5fwuSX1g87RD5aUJh-U4htAPPBkPWT1OqrJG8TvqS_LoMCTdPQPVU&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,RozNWdaPDnLkxzHk0PIIu7yOI0mq7J36SJtiYqHJSj83AaF6otFCOMT6S0SrKP0i5_HdKBbJFOfyhgDON-XzQcdSOvC2Z4N90Agiv9ZXimQSY87tDA,,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3xUOFzb6MCi6u1j2LRQL6CE5BWbu5eM6Ir7EsfF2oCaYUBtqnA1r3VDYEmgBiVBb7QypDxbOzY9knH1MLZZ3oikFpY_p69_553LHOTbV7g,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,9xoBaA3kGi2K8zQ8BuNLDq0dYmsnE1nuJw9sP_cDKKpUiUl1eHBvRBJF127g2yN9LNqLqZkyPVSq8YusAWP6k6MCI4_4n846oL1LY84Tow2q00oOukNL4Q,,&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,OqfGI8b5WUrx4fzs6Vhc7lLcb55MI-F2H4j9ic-9Fbs1_FhBfTLzdobmjbVxUHuNV9kcphYOQdlHoTf8bN7hu4yIeKsdhwLW0rasAMuEm4mWhNV8t2c,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,ml--e7XB5_TgtwWFh6lw23pXHYNiKpjFGKoUKmwrOwHyS3lnFQ78O6WUjq-OM1ITxXTrgD6P5K2E89TnqOgGcaOyn6aLaVIcvPhARMyQEqlFDI2mcnYwIfwADKB-&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,olHwwJjC2TUJmGxt3e1EpgRMfVMAAdmudKrqZ5ZByilJc51VAK_ZZHGL1mLqUqv59mtQqrIVOT0FHxcuVGnfmTsHrtP_Lfh6vEjgZmLr0Aa8EqAKSU0BDQ,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,3crdMkhGhlkrJXp-HsjZIEJOjIoajfvvI_g-uTOHgEc26-rm5spEwaDR4tTrIOKeN5HSQz3HfKcnanjUMfuquaMfvk7M7K2AtlUVxZdR9Q,,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,Qv7RvLuSjcsV2_72qHHLM6lmOI05RppEGGyR8280SVuKWQgOAl-EeFo8ZDeB_9Hw2f5ZgHbKDJbuB11rT-FBX6wf71ASJsIGNZYXzHn1HnNPy3pXaBaEgvrJvA,,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,7hPEKeOrrW0p5qUGMytIu7avFNFhxNhXFzKgo6a-hDqm4ky01dphgxAxTpF5DmP3oUrVV1uWYwMbJuUFv1IoX_o9-H7Y6xXWdj067c1A&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,dU7lk32qJcXeW-cUtM1E65BB47JiLhSgJYTqhLIX373Pj0miueYi3kN7MYtAU5zT39qOtzzAiHcBVYWsGDB3N8R31seI_xbm-rQzfd0jzb6V1rSLy1kotiI,&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,BtR0iFYtXaxmAlygBDvC74BLAVELzJivsFTs_YS41zS_XagPPAr0UUKW1lNrafWE3WxaZcW_S5WbNn583_uHQ7aLAA9vskUJfxaMwUrM_DxAOd2N&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,nW_o_ayhGlEOytZThvhNsFjqKDq7oEaYAEjw_jVKEHAnXglYBlSGSA3yj7wyG-2DXRXiRZSD11UwsKx7fSGPuHkUUSoULIYqacVjSrfmUuEfG_A,&typo=1&ancr_add=1>
> )
>
>
>
> 2024-01-10T19:52:34.824 [Thread-19] JmsStaticConnectionPool.onException:78
> [] ERROR - Receive error occurred. javax.jms.JMSException:
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(
> ActiveMQConnection.java:714
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a714&c=E,1,3FhMyZ8H2FihvFzilmw5MqmH_tkqp5vHbCOJsN5x6sQNSLyFELuJt_Ej2M_24CH42Xx0iQesALGDJK8HHdR_PDaLploHa8R84-uwfftA3DUM&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(
> ActiveMQConnection.java:735
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a735&c=E,1,JvpZWdXDc673FDsIBWz8WgGu29y6j8H0zAczvf_UXYyUAzEMKtvXJ3tMkleaFsrHU-QdlAUi0bWddjnikuurWEEMfZs0jIDWZ3jnrx2__Vbg93yZ8oo,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(
> ClientSessionFactoryImpl.java:868
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a868&c=E,1,AznnL1cSXDTbCj6iQ4ovEeP71-oopbCcH0laxV9dB-Y_kBuerNfeOAXRkRLybAqgqQBjJZKBNFEoveCJjTGncMCguHnY0hvsdZuKIHz6BEhvRX7N&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(
> ClientSessionFactoryImpl.java:856
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a856&c=E,1,vBewrTh4rfp_ezwl572jqeiy0VBShHwQJNRT99yv4B54VM5Qaf2hTHrOq33emXwk_WSFrcueJSCKI90AxcN4jUznfP1ZLzL1dJdUz01zK4vN&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:802
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a802&c=E,1,EY-ry2DzdJloXWtaemW-dWnRy1-rr2mQFFjoCbmWjX6xHCNZF95ldY2zUgfhOhSQFfzm7eYV0PdRUTOwJrx9jCD4U-B5ztFqWmvqEmJQuqdCyQgypA,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,gHouFFMo7plejV1zi4ureD9QEDkW-WjvytRlH6DOyEt7oMO8ULkj6APov-gDhxtXAspTlk7qF6RYw-RrCqfSrXlcE0Cm45J61qJHt5Yc3zuBJJyEX8A,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,17MGSyeFWTwZKhb2DCqWeyFvdlinlKAunc4JSHJltjjiaCeT_1UjXFsTMtEiF_NYIOMI3brJbPkRW1QkJIhe_Br44jy1od8-BI3l37IfFJzZbcculrc,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,F6NyfiAiJI9fxHnOQnnpO7dLyoFavqwnNivvLL6x2LO54G8SpNhXjlUN748SkNWwe_oCD9JhI7u_BOk1uNbwSTYV0lXRyewBrwvvysSEOSCggiFCFRgjtA,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,TnFdrKl6nCZv9PR6lSsRZ2YN0r8LqpGQMRo3oHlgyE7_AgUMzrW12R4HeB5JhX8Akv-1iKeVGrnTLV2WeAbBOCM-fukhfRPsdUgq5pwe0yU,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,vAtPtyOuzaufOELKjMWnOBMrJEOShH0EzH9G3WL8gak4hWHmnVWPL1W7c_YnxRH2IerBQErg9VJApOiiX6TR6tcPau3xqRHrPyrsH70KHg2v&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,pak9EX0xrMkRTN71voUljZIOmUvksqdCCqWdwh8fq2R4OZXUQFIHNH3P9j0t-wpQStfSs_ZsT0a9BZYopaoyMSnMkN3xqvXn8A9dTehv&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,acJEflBFObRkFNBskLYk7AARfS1_CzZH3yFGYB9r_Ype7n0DAWduZhGpmImVc9k1pTnmFoA8L8UukMVvWBinxujU6r75ipmzh4I_cNQ5Gzmd97g5SAU4dRsf3Hw,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,aM5tVNMvOtLUxuoV8Dez1Y7fAS8KwME3peX3IqP_QPViIWgkZQahJW_1BMBQUo35u5h9hGJ3XSyDgkECwWOGDn6a9W-gctjh8WvUKx9yQ0sSLwheMcs5mhR2yVM,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,fUWcgLX3hupjGCjPYEByZ8xfzFx4iN-VnkLB9MarNu6OEOfrwnCkvJvtu6e-ZV-FREp0Ssisi_fezDu18eP9adS4mmcPPlJH69vnf0YDVg,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,KuAjbB9H9ddyi6unfXrVdNXylpB7l6qLIUfJARScyUnc7iH1GtvO0s1MAtB9fwkdqXq8idOJYZs_rg0iU-sN4aV24ID0Os5tsyvB34ba3ve2D3diROqixzkQuw4,&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,EcYd8WT9GtMN9IXo58gsQpj5uNwcdCRtDfa0zG9t2jpvfdaSX5r9vLey2icJ-4wo3EyBMyUQzwyqYwC5BMfe4yHGjyo7h8jxmjcEKig4V0KyVa_SVq58NBT8&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,_a7OW7NorEoCm9eOnYjCi6TFl2__or_ickK-DwP4rrMmRX2Rj_TKaxpGjSAwCJZasllt1gefhShoob68MthMyp6bxv9N5fucG2HiF0g3yK4kqHTZjiFn7kxZ&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,-ttWHQzQDZ94E-HqKtYTKV2nuApkTk60Zy25iWFIykqfNe0o2seEul4UE_gETZr08B40bJjmZKMHq_-R0M8bQ8Fbu8twSkWx9MPLFUhc&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,624T4PuCyLmsS-UG0IoXRx3MMz4DqnU_1BuWfOz973WSpqfLsFFDAICu75KCoCwAs9O8WgKTEcBVqRhhf1JUqI3Znxem3LsG7Ml0Td3DnJJUoGYR0BE,&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7UL50c_ahB0IhSHbe9OlGABBKKE4OFyNM67SjmNrSo_LQhfViHAlLgvwdlRpbmDdymI5VaeOEklHjvlweLBz8Ske6wjc8J3CzM05VYenh6DzNiI,&typo=1&ancr_add=1>
> )
>
> Caused by: ActiveMQDisconnectedException[errorType=DISCONNECTED
> message=AMQ219015: The connection was disconnected because of server
> shutdown]
>
> ... 7 more
>
>
>
> 2024-01-10T19:52:34.828 [Thread-17 (ActiveMQ-client-global-threads)]
> ClientSessionImpl.acknowledge:1225 [] DEBUG - client ack messageID =
> 10054445
>
>
>
> 2024-01-10T19:52:34.862 [Thread-24 (ActiveMQ-client-global-threads)]
> JmsServerBase.logRequest:117 [] INFO - REQUEST IN:
>
>
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,01YVdf0IHIl4w2xIwUh5i7ToRiQfv6TxT-FRWSNW0n6e2aucHkWVyKxkYJTpG1jt7_EUCuQVS0NWGalDyf11AUIb3qVPj67IUtjJB484IdLKrHgbmSd-Q8Vg7AKZ&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Tuesday, January 9, 2024 1:57 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Both the "replicated-failback-static" and "replicated-failback" examples
> run without issue out of the box on my machine so I'm not sure what's going
> on with that. One thing that's odd is that out-of-the-box the line 122 [1]
> doesn't match the method call in your NPE. Perhaps you modified the example?
>
>
>
> Aside from that there were a handful of questions in my previous message
> you didn't address. I think the questions about thread dumps are the most
> pressing given that you indicated you're deleting the pod. My guess is that
> the client is getting hung in a network call that doesn't want to return
> due to the pod disappearing. A series of thread dumps during the minute
> delay would confirm or deny that.
>
>
>
>
>
> Justin
>
>
>
> [1]
> https://github.com/apache/activemq-artemis/blob/2.31.2/examples/features/ha/replicated-failback-static/src/main/java/org/apache/activemq/artemis/jms/example/ReplicatedFailbackStaticExample.java#L122
>
>
>
> On Tue, Jan 9, 2024 at 12:47 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hey Justing, thanks for the help.
>
>
>
> We are using version 2.31.2, which is pretty recent, if not the latest.
>
>
>
> With regards to the log traces, other AMQ threads show similar behavior,
> and our system isn’t doing much until AMQ-client stabilizes.
>
> I am running on a k3d cluster and the failover is triggered by me deleting
> the dm-activemq-live pod, which k8s then spins back up.
>
> Also, our slave config does NOT specify “allow-failback=true”, so the spin
> up of the new dm-activemq-live instance does not trigger a fail-back, which
> is intentional to eliminate failover-failback chaos in such a short time.
>
>
>
> I am trying to setup a simpler test case by using one of the examples,
> artemis/2.31.2/examples/features/ha/replicated-failback-static, however, I
> am getting this NPE on line 112:
>
> Caused by: java.lang.NullPointerException: Cannot invoke
> "javax.jms.TextMessage.getText()" because "message0" is null
>
>
>
> The NPE seems to indicate that, you cannot call consumer.receive() again
> even if NOT acknowledged, which seems to contradict the comments on the
> code. I have run into this same behavior in another example as well,
> artemis/2.31.2/examples/features/ha/replicated-failback.
>
>
>
> Can you please confirm the expected behavior of the example code in the
> 2.31.2 version? I can then modify the code accordingly. Or please point me
> to a working sample code.
>
>
>
> Thanks,
>
> Lino
>
>
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,lIOPl76F2R4EtenIAEJMeQ17kfmHQ6buC7G1W5TRefOcSfxvj_-3dLXV3MKDkCTkac_65Rw0SYmof66i91zGwgb_SS1nKaBpao_Lw7JNNk1mkE33&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Monday, January 8, 2024 9:20 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > Now, the backup process does start processing msgs right away...
>
>
>
> What evidence do you have of that?
>
>
>
> > ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute
> apart, before it stabilizes after the failover and is once again usable.
>
>
>
> The log snippets you provided show a reconnection attempt back to
> dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to
> dm-activemq-backup-svc:61617 both of which fail almost immediately. Then
> right after that another connection attempt is made to
> dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e.
> from 20:03:44.874 to 20:04:44.933). Are there any other logs during this
> minute? Did you happen to get any thread dumps during this time? If so,
> what did they show? If not, could you? What was happening with
> dm-activemq-live-svc during this time? Perhaps there was something weird
> going on with the network related to whatever you were doing with that
> machine to trigger failover.
>
>
>
> You might consider simplifying your test-case and then adding complexity
> bit by bit to see what triggers the difference in behavior. For example,
> run your test with the client and both brokers on the *same machine*. If
> that works then move just the client to a remote machine. If that works
> then test the client just reconnecting to the primary broker upon restart
> (i.e. no failover). Then finally put all three entities on separate
> machines and test failover. Hopefully such an exercise will give you a
> better understanding of where the problem might lie.
>
>
>
> Lastly, please indicate what version of ActiveMQ Artemis you're using.
>
>
>
>
>
> Justin
>
>
>
> On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hi Justin,
>
> I’m working with John Lilley on getting an ActiveMQ HA configuration
> operational. Our current HA config is replication with master and slave
> instances and with static connections, and it seems to be working, except
> that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect
> itself after a failover from the live to the backup instance. Now, the
> backup process does start processing msgs right away, but the
> ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart,
> before it stabilizes after the failover and is once again usable. I have
> included the relevant log traces from the Thread-0 ActiveMQ-client showing
> the Connection failure detection all the way through when the client is
> usable again, over 1 minute later. I am hoping that, based on the trace,
> you can find something that, perhaps, we should be doing differently in the
> client side?
>
>
>
> Here is our Client connection url:
>
> (tcp://dm-activemq-live-svc:
> 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
>
>
>
> Should we also be including the corresponding connector names?
>
>
>
> In an earlier email with John Lilley you recommended 2 brokers using
> “shared storage” instead of “replication”, which I will try next, however,
> since this issue seems to be on the client-side, I want to make sure our
> client-side is setup correctly.
>
>
>
> Thanks in advance for your help.
>
> Lino
>
>
>
> Thread-0 log traces:
>
> 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-live-svc/10.43.197.248:61616 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=a8a4a77c, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> ---
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and currentConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.197.248:61616
>
>
>
> 2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.11.76:61617
>
>
>
> 2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> ---
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and currentConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.197.248:61616
>
> 2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = ced14fbc
>
>
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.11.76:61617
>
>
>
> 2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
>
>
> 2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 10, performing as a failover operation now and recreating
> objects
>
>
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address
> activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,hqiYs6vnJzfq-X42G9W9ohJtYgn9LRPR-82RHxMyPnKUD5xBwCbo8W6b-PegfmAGwvEENlDO0k2IkoGFFegGxR-H6J72UHHlpEFbGtJzhCF3sucQ&typo=1&ancr_add=1>
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,KX9atZvl9XxpGPWobW9hXsmaLrVekfw2221zvD0PJ9Ik3fEQVyHKcZZCe7MQRciamxD-T-WDEk5Xa7_cAt0PSHwVW3jG7zzhq8XH9XIRC8EXmg,,&typo=1&ancr_add=1>,
> needed = 65536, arriving = 0
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,ZOCwWoQv34cYLuw9hZvJzBHfSrX6oQeQ_3-SetFMJa1hRC6lXgiKsAElOhfZ_vfnBhRGRJPZjSOqHIbwWO2ltSYYY7ElQjyItp0uDo79_jaDkw,,&typo=1&ancr_add=1>
>
> 2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=a8a4a77c, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> ClientSessionFactoryImpl.java:410
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,obbNTaFqn6mglgyZXy1S-0nBLNVtEYlkZBwAEKSGDGN0RuqlR_-7fIQPW4ELNqVFJdenAcKh65C3yQ2b_Lef7W01Tst6ZTrmI66KfHLae572jXv-HF3O&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> NettyConnector.java:835
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ytVVTXzfq-J1NXl2ct7T_47F5EbfZNQTXP8dlqw720EiCe32c6CLwlpk0-v4MzUD61FErq0nkRTalFpcTtIN8TDeBvz70rzb__TyeIDm48ZKfFU28qQ,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:666
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,KQYUuEgOaR6V8a_ZC_xBpKZmDneB4LTL1cQzn2Gqtla8ntHJ1fgRs9KelLgvbMd9zvBTg8maXUerYZYQZa-DlfK7fon0JZ0O74PNJ1SOCHqIPN4fZQ6Xv731OGOD&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,OSVh6NSORxz7qZ6eIuUDgWskoYfyXT4NNdjJYGMbH7HeWEUiymys2DnptjYKdmQfqjfHmI-egRZ31yWwd9MKzEd-gbbsuTUsqn_46niKaw,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,DM2boEBGg44zzcd0m37ITkOnfDrkqLE8ZSFGJpEI4AGo2GyOI2yKX08AEkKHxw_UnDMF1vccYEvESFDXjXBBjjIGO5TEsMLWZHbJF3AOIEQ73uc,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,wipgGeA7UVZIy1uuqvuxJEx4BA8TvYfSYmO617IgrvnOo-z7sn1vUjXFqIEU9M1HmedBVoHxB_LpvuEZ9p_biioPM8V-V9eS8i8oST46GvBhZZIc&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,0FNTjscT_d2CNIid0J8d_x7C7Peb6_neE-a_MZeKqkM1o_sPGujvpkXuiLv6kms9wSFyYqSux86C1MX2R8WKM8IZBsjKt7sCs03U2lRYlVbw2eTDg0ihSEV1nA,,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3yKPqxpIi9n43QJSCQiysKGlEYttZ5Dmuirc7jUAWNz6orLfIUCYwdDEJ3eU0QiHywsNAJdWqrvnx5rZK6-8TTJb6o8tr1WSvSL7omLF&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,6B4qLrj7q9DzDb0Trp9xgst7cOBhbD4hcEcGyPFul5E_mh3_oGmColB9YQkQTB1gJ6x5Mcm0VwvBLddpDpfFkcZwlzb4YNR9Ym5fcDclFW97pIrQ9oG4&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KqR93A_gEw1FpU7__PsYjSkOC5LH0zFTSKu5K9YlCdVAz3h_c9EkBndO7aliE4RSbK_qslEZp0b-YzJfCOxmB22jAfplsYxguMl1qjC0rWpNjfr9_fh4RjNL3AOK&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,mhChlKkihYjDw_Kk5SL1d_4eb7zhBlSzri_nwjOUsrs6MAvEfDAAjia07yTMUU6CQqc0WbDS2Fn8HJJI0UwJh1N7SKiz9dsP1gJLmgY66huLhXZE9c-3ECGXyIWQ&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,nn8IDmg6XQJLLXzH7kunXDNZa_3A7fUlnHdRCLoha7gQWJO1QeY32duGFAQMYaFHsfIbHbv142mei8HfCqRwrpuUfikYxLkFoXaD3pORZb9roBJs0H8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,DPLEvt-MPmmzv78LEkeXtYHM2q37m12TxX-nqjuIJurQdeXydsSakco9t_O7jgBbQ6y115xxdDb8IonFZbM86zgRDLcvUmeOSJuDmZybD_6JaTX0zpbQ&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,1vNFV-mDaLK92lJo0Mq6jvkCDzuwrSHT4lp1nB6KTzOCSO6comH5xNpEcvtZ_9pc0cQCD2kgDfB5xdxg0tAX2dF5oG5DHY6kxWXOBLnuAwiz1BkEFhw,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,aLydz0SGsclotqCro2upcTg44B_Ij7ckjApLu4B5JUwZKKdMo5z3RyUR4ikHFY8L3t_yB87Zp5atGRwoAPA1L9Mc9c681UjFrG16n-udVjeHtxQNxx8V-Nug&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1mZX-u-XHEZ_e4Ut7YETRvWm7U_qkC-q9QaPeHah4K4yNl5foyM1VSExCtYiisknkelDj6_K2MUHGc65Kd-6-JG_TXLby0OPWodJVjX3BtKaDNBUzusgpzT6ajL7&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-nbEPDxI7J-DFvbJfVVt2Cs1Vi_hN5bw-8vQF3SlM1nthw3VbJLWmS1h73cupUxqBoHCImTzDqI1m5gutx3ZriLpLpU7IqutHCqZuT6CTVWACPtj&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,-lUcdhDzaHWdAWLCc0IxWCFkKtKHxoEd64GWhzXAbeuG_Zm9mvn624BY6uS8MgQn-s2TMF8PbEPYVbOOy6BJuI8ByeiuAF9Y8f2I5wib&typo=1&ancr_add=1>
> )
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,T2UV-1dji1RODRcU4Ff4j_IzfTH2n47BatzyLZcEOU8VROFr3GS8re7CjDcnnHCSaPEtR-35J3z30dnYcz5K66rVsMVxgDiyh0FvEDKnN67naHjYbn0NpzAc&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
RE: Looking for HA/replication boilerplate broker.xml and advice (continued)
Posted by Lino Pereira <li...@redpointglobal.com.INVALID>.
Hey Justin,
I did finally get a chance to run everything locally, and it all works smoothly with failovers in both directions and the AMQ-client recovers within 2-3 seconds and with a lot less log noise.
I did not get the thread dumps yet because I’ll have to instrument the code for that, right? Or am I missing an easier way to generate those on-demand? This is happening in a k8s pod, so not easy to connect with jconsole.
Thanks,
Lino
[rg] <https://www.redpointglobal.com/>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>
Sent: Wednesday, January 10, 2024 3:02 PM
To: users@activemq.apache.org
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> So our issue seems to be only for failover from live-to-backup, consistently.
As far as I know the client doesn't make any distinction between these two cases so I'm guessing there's some kind of environmental difference that's making an impact.
> Is there something that we are perhaps not doing on the client?
I don't think so. The connection URL you provided previously looks good.
> I will extend one of the example codes to use threads and sone of the AMQ-client wrapper code in an effort to isolate this behavior on a simpler code base, as you recommented.
Before going down that road it might be better to inspect the ostensibly low-hanging fruit first. Did you happen to test everything locally? If so, what happened? Did you acquire any thread dumps during the delay as previously requested?
Justin
On Wed, Jan 10, 2024 at 3:04 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hey Justin,
After starting fresh with version 2.31.2, the examples work out of the box for me as well.
And, after ensuring that version 2.31.2 is running everywhere, client was still using 2.30.0, and refactoring to using the recommended shared-store HA, our AMQ-client is still behaving as before. That is, failover from live-to-backup, client recovery takes over 1 minute, and logs report what I posted earlier. However, failover from backup-to-live, client recovers within 5 seconds, which is the expected behavior, and our system is happy. Please see new log excerpts with more logging included for the 2 relevant threads.
So our issue seems to be only for failover from live-to-backup, consistently. Is there something that we are perhaps not doing on the client?
I will extend one of the example codes to use threads and sone of the AMQ-client wrapper code in an effort to isolate this behavior on a simpler code base, as you recommented.
In the meantime, I thought I’d ping you in case you can think of something obvious that we might have missed, especially since we just moved from 2.30.0.
Thanks again,
Lino
Our versions:
ARTEMIS_VERSION=2.31.2
JMX_VERSION=0.18.0
Logs from failover from backup-to-live:
2024-01-10T19:52:29.588 [Thread-24 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d7ff7885, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,O0LJ4VfRCemWEaMLULajHsgUo6YseAVDyBlrTm2G50vM20WhlTUV0ueY1DxMRAMStXAoc6wy5Xx3x1vi2XsnE5F7_iNbL-MZ03M4IS9G&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,VdcejKqF9B67eWPDv-PP_0gR6PEmUxFYHHtW8kyn3Gzg3q-HHzL-Lr7oztjj2LQumPJWO-1KF4OPavU8c3DW4zPgrUsidhnnCpoMUUn_&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,xKCDuAN84vJBjsCp75epCz7V1lR05TFhETdvpPcKntBTPnQk0zuuQSei7-ly1RPUUsHDP8ZAe-IevqwGtTGfEB_yNzw90QcOpEx5BaXOW6E,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,pq7BenEb7b9i-FYz1LnTy-gHYDdV-28FiSv3pkV_i-1TQZtRxakbT-rOKkgLgz2IDx875Dk5EE_OrwsUcew9TIsvrcZZZ9ZIlwboxMEq9xnR5odpQL8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,8mmrZ5alRiVlyEshqjlGpedpt7TuUVBJOy8DDm2qHLGoY6v3pR76llV59sMsYwy2v4rwLqiD9Rcf3Hg4QSCZo7rdO1rQgMNapCKji2jg7T4IJOk3OAeLnIs,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,RLTYzCoyknTJ1GB9szMq3Jg02QAnt7juedv1HCTYmmA4JU1DO6MlJhkvzOTBqHbeeOdKSvwkualBrhGt0fUk7Nl5YLzlgfepqTWcDiBmXQ-_xSMcAHPw&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,iYtEGEWoGf58Vq3CGAxBrH3iWZ1jWm1N1b3AOFPNlx0kCahcxHEwXT7ZAQ_h3ynXTeKJSkvmBiGUkbqE3bH7tGw-Gn8VV7R7LaCkxl9Dch5I4kQDfJDKSg,,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,kH54RADAl8RcHDeCxJW9b-P4YJJFElj_Nbh6mHtmwlFdP2tKzzbK3wkAXg9tVqhIlsbd4gr2LJkn7DWZ4I-J8WVE-7UovVbVakuQTTxc2pbR9vuIDiEFztc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,V9-XUYgB7GoqPyGhEdE_CmtNood-yvXw6P9wRIqQOSYSWuTfiF1bhonQejBQfgJdHq8k2TsE1RAvcSOp_WzBE-ybl0l6rc5pIuMGdC1LUITpdk_Iuha7urSje4Y,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,RNgnFH7dBkD27TX353Mc-kqneOLcqFXEgpGywxg8VduTTYNyYFt5VqkKfirozQw9V88WOI8XK4v9dvrtPIl-3u7cp673GvKxq3L-XgqypuDtSnkRIYnh813U&typo=1&ancr_add=1>)
2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d43c72b6, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,L97BTn6AFLbUpYgHzAlQm9pDJ4Sn--TlGDpz05xOE3xd-qn9krfLJ6JzGTaiSpQwuVjuQ8KUPfUSW6N18pgw44-ay7eqYb2h0p3sxehZeXAC&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,PiedcDlDv-nxD-EF13k9j-sagD4clTOVvCs6xyDQvnrPzDUxc4xAbnHg0d7hnqeG4cVJZaO42LJvoNm59E3SmClgLAft-3N7F898HO6Obbkm&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,G5SZ7tGYpGBQzNRxdmIzsp4hWlUsmbY3_881k0N5lFoFBwuHt1dMCkKHI8wiZRGpS6qDY_UDBvDmKVQNRIHXIIyg5Df2nnWDCTLnbvWCO20fW0PC&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,MsagygznR7ie9qgJLlMogXq8ex-X4ST-fg_vbVbgcBanXur80DVykS1ftnzq0e4bWLE0k8EiNmlhlezemqTlBEncWMJz-AQAKdTxEk4WyVY,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,n7l9CUpZZbFH3ca7Iu2wxM3kNfged80mMRcu0Vn9wIoZz7aKzCP3c1OVR73iqF0oLbir7CjMHKAu3sfrR2cLs-EP7orqDWQKQApkzTZra9CJag,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,PePvk640nUsvwJWNJY39J2R954rSgiFVPPSQLOsjYskMxtDnDVBuzQiZ8ut5SAPoqzV9uYW6GX3ekOAAoou9eu6VpHa_oFc2-7d27W00VBLU_o_Zv4wEUw,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,REmfZe55ytZRsqZvFkCNsYUcRO5Yd8PZwIlQfGystV0qUeF2mXSYaWKgleI098Ac40PND4ilF-ukqEWDY8N04AIRQgGgKGpG036xx7jNSc78YS6UbKWiMNUHfNc,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,nnqfN76WKNs0ZYhVPzea17QO5aWkwmLqbfYrmQJ_qGQIgTzCL4PSXhjU9jy4mhbCUfl53cwF4ISYabr_zD1GUvCgv5m4dYW1_OROCBpe&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,vlJ5vYZAfrlVp9kL50a_jwWhqVW8-LMUgzeBb6mn-udcS_fMR1yld889bXgOXTMDiutFqKYX9XPh0RtxeXRLtnN31WqeOlH_CZv4_ZyhnOwXvm4J&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7Ba-VAkMm3x0E0f6F1f73TH_y9Ylo03hx9DEz4kqWzvrXy0gAv3D9zIM35eXIaB95UFxo72DkMpb0nKW_a-BIF89JobbRqjyVNLdC5K3L1EM520B&typo=1&ancr_add=1>)
2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:29.592 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:29.594 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:29.593 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:29.597 [Thread-6 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d4cf19d5
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.600 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 7b2392ac
2024-01-10T19:52:29.602 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.602 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.603 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = c1fb4c82
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.610 [Thread-8 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 06b7b3a3
2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:29.611 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 65680be4
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.624 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 97a109b7
2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.625 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:29.629 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 77b2de2f
2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:30.612 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = f92d7cd6
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.613 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 1d44895d
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.614 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = e8602acd
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:30.631 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 0a217bfd
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.632 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d2aa00d7
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.633 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 9a4f639e
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:30.634 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = ecdb1cbf
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:31.000 [Schedule-Runner] SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service as system
2024-01-10T19:52:32.001 [Schedule-Runner] SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service as system
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 2/200
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:32.116 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = a6b3d488
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.117 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = c0627cd9
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.118 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 71523eb3
2024-01-10T19:52:32.119 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.120 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:32.134 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 2/200
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:32.135 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 68efd0dc
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.137 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = daddef73
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.137 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 84813ce3
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:32.138 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 533990e9
2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:34.370 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 3/200
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:34.371 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 1d8df71b
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:34.373 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = de20e193
2024-01-10T19:52:34.373 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 3/200
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:34.390 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617<http://10.43.229.161:61617>
2024-01-10T19:52:34.390 [Thread-10 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 4661fd5d
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616<http://10.43.45.150:61616>
2024-01-10T19:52:34.392 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d1c14a74
2024-01-10T19:52:34.392 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.404 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-10T19:52:34.405 [Thread-9 (ActiveMQ-client-netty-threads)] ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up backup config = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc for live = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.405 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 38, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.409 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-10T19:52:34.410 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 37, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.411 [Thread-24 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 10, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.411 [Thread-11 (ActiveMQ-client-netty-threads)] ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up backup config = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc for live = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.413 [Thread-24 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,S5999ok8Om6JjNfBLhC0R2hYdl3dGqPWsLpis0MTx38AhKXPNqMMlH41RrDz0UZsxZzSZbLvUEOYFEa7hwM4N7gAZfCS9udcqSmOwhLg&typo=1&ancr_add=1>
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,CF5ZPcR0aBH6uaRjLjSeWoBKLnRwUOOgvrD5MzWuerKGIs1FEa5vZF8rdF1kWSKbLwGQlm1IsOeyey0f1t7eiA1ApUgs1EDhL7ilSV0c6rFxEanucXI,&typo=1&ancr_add=1>, needed = 65536, arriving = 0
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,nl9VlpvL5Lx7cxWUON3eMxxXV1my7fPq5WQLaWD1ozszhEPr1wVfqCwFilScj02MzRACloIhLZ8CgyMWoa_OqCu-Tf4folOdvKvtooRq&typo=1&ancr_add=1>
2024-01-10T19:52:34.415 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d7ff7885, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,awSP7U_0_MIm7Km9lwe59SrlpKsd9iUMA2y266CCeNZS6GJuEExdDYVS6EVMnlmPlVN0W1aqSx2Rss2Dm_YIM71s5j7OAUz6pAdGaTqnU_h2RyaoD6KyuoU,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,8gAKfG0MZn0OMdhCHJ98evOezuGoCS0mNbGWplSwawxhF0ed3tPq9vKWoWUbn2xmoLJ34skkCTpI2tUuevSCeVNT00tNpFlOJlyAjGWu&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,tXm2hjddgUkJac0QpMg4nVt2gwg5e3ORdql8K6xsJlBZIxph-NuPt1Ks2iB0ZarzoFELPAuAerVMiQpEU3nkq1JiqLAju26hE6Y2c3MwmKRTj66O8tsfQ-WFSh4d&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,P0xsXGcezpL-KzlLSOt_rfr6b6RF51B2hthJzXMGD04QeyLyg6AnMvF4M8ZOEwF90ojRdT3CtOy0x9HBSpGK_mmF_6Zl7Muz2TKrZRLAxQRxhV5XnvYlwQ,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,qctWvKkM1LAyegiROEl3hsiEN3dPcBcH4jlkBConOP-G2cgGl6X6XG-kdeQuBBOC8BmtTf-VmlbpJTjC_wznn8I6g7pLOg1uof2v0j_r3yyK7U8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,KRtNFryEe6jhqdT3of2FZC9PF7zGlYTxJ3KPFwO6TViZkyaHiybwHP3JmOhQ4F7yZtorUVRe5phyCswDWLc8lIFBVowIJygjwxLmO-F3xJSonSQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,ltH-LxKKSiKev4cs4CgS53B1Sn1z2t9IkxY_vjCKeoXMalJfF0yTujWPogG1a9qkWV5DrAT_qCTk7MdQa3LpkAae5yHTe_19fREF16p6vMMJqGXqLjw,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,66enqLb5-TVz_cB0ZCkDW4SpX4c3v0jslINMsxhHSjXfBWrIFoM8PxfSS7_VWmc_MdfmjPHIKcLm4zfcCe4MggwEWLxB-RjiUMqnmc8ShDJLoit5&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,xM_aKOMjg39ORE7gTvaGKP72ytDNvt6tjanfNTzekduYVGBvn963oijBiSY1JOQeorfgcCf57lVZSYIQ20x8sOFAH5zNexuTI0rBtrkDrtcJpq-S&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,Jr0OyBaxbJX-r7nzGvHtUkNxDkYucNHeuCPW_Mqw5I4IS0dve_XCjayhv7uvFQTdinHTsHqf2ZtZAn_qz1ZnE04PEsE0KWGfHQUj6N5WIKcNhyQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,fTod1kMxyNrLe6wmpPVQB1vco-7IrNIs9Dx8h9kWYDCH7xYH8V5kj2iao_X4D62DX5S24Xi08LA-xy2zHwj-LIzSKOA38iBBAlHdMD7qu3lK&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,37VGZcLM1KBT3tttPbLrhK822id0s0HCTimMYb18Jk-8bCbWxvPl2N9_yFFLgXa5pC9DXbKX6c8DMHolUFme8KRWVQdBx_lbRwtpLIOp6O7uiuV0fttkzvI3jxdO&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,zIgfFXrrGxOB_lKTtscVc_jXQlq_pH7TMnaK0IkixM4MVqoSgc-Wp5VV-QGQWmUtW5dggcgtYpRa7OoRHCBvDew7fxYJq28eSZVguCiuBiH9DzdKTHAXC5o,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,b6YutPqRZaL3jtcBxfqj_SiU4M_x_Qx-GeSmBfL7c6CG24IUL7AdjJxpyosXZtCtKco4bSDRxOj7HJn5CWgbgbPzJ0TFuXzJjs7VRUKVhg,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,xdw7r1BqGJuJPUUyo2a9wj_y2KKTC-j8c0DAocpHKzpZZaVOgPcxdng0Xi_JMKvlGYmcUvlTfZMm2FYQ7pTrF20xWUreNaRQqtU6rlWB3kkuPmMNBALyTBs,&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,4iCAaiDF74c7cCKTdUIIlT6Qu6BmbQItK-gaAIrQtpNdie21ccXNS_dCydALepUtQNTgsVsVUs1NavHOugg0IlFK0CjTj8TKnThm55ffUeOx8zHc4oZUYpE_LRsz&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,qBGyQwtcrkAjVKJZ9cZzaW1ivhGQgGfhpcJUS_ww-vfcQVNfD9OjDwDX1XjpzJWYpo08MqSOrKLpgO8d72TBVtSGg2OQ4AmggCv3IUgKeA,,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,Mm6fpsoFsFLUtanrkJ6cqlk2y1zCJbPDIfqxsd0pS_sfS05oTAHBNIXIUUd4iA8HJY0guikOejZmykugHqKeopxLvJE3UmpX9YX1Y3Gu2ToK-B8aRw,,&typo=1&ancr_add=1>)
2024-01-10T19:52:34.435 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 54, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.465 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 98, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.474 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 20, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.483 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 128, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.495 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 158, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.499 [Thread-4] AbstractProducerCreditsImpl.checkCredits:184 [] DEBUG - not asking for -1 credits on activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,BYjREAStKM-0e-e5OOY86CxL1g9WrAP01m5ROYtsGCMHTh3wox28vU8_r7Y9e7QwRiDtJ065Xst_zM3u0oFx_K7Fr-ZkN1GBxPR6aWEKrm4nQA,,&typo=1&ancr_add=1>
2024-01-10T19:52:34.499 [Thread-4] AbstractProducerCreditsImpl.afterAcquired:98 [] DEBUG - AfterAcquire 333 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,r4o5lwz6oA1VbUpicHhTmEpC9U7BwWBMw8ZYMcrl4g-jVFykL22Cb2WJIZLOqv-hLbxRqgThYlmO40Jo84JsPHe-IYBjUU8WMQ-jy1lZqxq3YQk,&typo=1&ancr_add=1>, pendingCredits=65203
2024-01-10T19:52:34.506 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 106, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.680 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 155, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.683 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 21, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.684 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 129, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.686 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 57, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.687 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 174, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.688 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 176, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.689 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 56, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.690 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 108, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.691 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 156, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 233, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 112, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.693 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 187, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.694 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 110, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.695 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 144, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.696 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 235, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.697 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 28, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 47, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 30, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.699 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 77, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.701 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 65, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.710 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 70, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 34, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 159, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.713 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 76, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.714 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 119, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.716 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 59, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.719 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 87, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.720 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 80, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 166, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 149, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.722 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 137, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.723 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 163, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.724 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 179, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.726 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 181, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 17, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 58, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.728 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 32, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 49, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 234, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.730 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 143, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.731 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 42, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 136, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 178, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.733 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 55, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.734 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 95, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.737 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 48, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 16, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 146, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.739 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 22, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.742 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 43, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.743 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 130, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.744 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 36, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.745 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 60, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.746 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 117, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.751 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 14, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.752 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 81, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.754 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 93, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 138, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 100, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.756 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 62, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.757 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 125, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.758 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 15, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 99, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 44, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.760 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 172, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.761 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 84, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 79, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 122, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.764 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 64, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.765 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 88, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.767 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 151, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.772 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.773 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 96, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.774 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 27, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.775 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 73, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.776 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 90, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.777 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 134, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.778 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 35, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.780 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 118, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 186, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 41, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.782 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 31, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 150, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 145, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.787 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 127, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.789 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 132, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 170, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 142, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.791 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 53, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.792 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 86, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.793 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 147, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 115, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 12, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.795 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 135, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.796 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 68, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.797 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 103, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.798 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 141, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 165, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 18, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.800 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 33, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.801 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 116, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.802 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 124, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 26, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 50, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.804 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 75, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 85, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 69, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 97, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 101, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.808 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 153, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.809 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 107, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.810 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 175, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.811 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 66, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.812 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 111, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 152, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 45, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.814 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 61, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.815 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 113, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.816 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 105, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 162, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 171, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.818 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 173, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 184, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.820 [Thread-15 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.afterAcquired:98 [aoy5vnfnjetz,6ek4wlg8c0ad] DEBUG - AfterAcquire 781 credits on address rpdm.v1.authentication, pendingCredits=63974
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.user.group.query
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.user.group.query, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.user.group.query
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.canvas.state.query
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.canvas.state.query, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.canvas.state.query
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:184 [z7wx0g93sdr7] DEBUG - not asking for -1 credits on acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.afterAcquired:98 [z7wx0g93sdr7] DEBUG - AfterAcquire 505 credits on address acf62840-864a-4eb4-a902-4861db94eb93, pendingCredits=65031
2024-01-10T19:52:34.824 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d43c72b6, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,dWlUyTFJQxRIqOG8poa5pxge5mYVJ48dUXfACv6LUtKsZzFSVGKyZagOScZPGCnB6cxsyzdobSyk8B6Bwd79504jcLncGDwqMeY-9K6dpOs,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ZNE9b4nkKfo7M-_v-eXIZ86i5mBDuVpXhcuhH7d4Z3TXAYIWBA7iz2F7WJKgkGmWz9vMjcTKd3WNKgAqq_OrNuwfMP9rqwnZZdfDvOywCznI2b71&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,ud4sg1bGnGSsxR72jGNWKmRRRqC9RhLjc5Fa_mAH3joB4qQCFRRYETYRnfp_427BSgD3-id4mG56z7G-3qi9hl8xOglhqXamz4f_kpydTW4Drw,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,5ea91cHeAd9MIKFvy9sbzhVk4n_LaL3aA10YRDmCls04v12YLb2XMmXw6eBUcLVuIE_Z644AmEt_e0ljvQnrS_I6HavZOLh4zjfiHWUvesg,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,jDru8ObAZFnt6UJuRwWa5fbG9hi1-FGT2jjgjYcb0j9MnKP4NE8WTbKBHSHyKvKnv60VBWqMlES8saZuop8zduWr_U-bg4uKdFl24xth0Wc08-CvUBpd&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,FCoOt1eEzJLmEhVMnIIHAKa-kTFceMfQD6Kcwc4ZSxQI7u6ROHuAf16piW1PXfz5fwuSX1g87RD5aUJh-U4htAPPBkPWT1OqrJG8TvqS_LoMCTdPQPVU&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,RozNWdaPDnLkxzHk0PIIu7yOI0mq7J36SJtiYqHJSj83AaF6otFCOMT6S0SrKP0i5_HdKBbJFOfyhgDON-XzQcdSOvC2Z4N90Agiv9ZXimQSY87tDA,,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3xUOFzb6MCi6u1j2LRQL6CE5BWbu5eM6Ir7EsfF2oCaYUBtqnA1r3VDYEmgBiVBb7QypDxbOzY9knH1MLZZ3oikFpY_p69_553LHOTbV7g,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,9xoBaA3kGi2K8zQ8BuNLDq0dYmsnE1nuJw9sP_cDKKpUiUl1eHBvRBJF127g2yN9LNqLqZkyPVSq8YusAWP6k6MCI4_4n846oL1LY84Tow2q00oOukNL4Q,,&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,OqfGI8b5WUrx4fzs6Vhc7lLcb55MI-F2H4j9ic-9Fbs1_FhBfTLzdobmjbVxUHuNV9kcphYOQdlHoTf8bN7hu4yIeKsdhwLW0rasAMuEm4mWhNV8t2c,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,ml--e7XB5_TgtwWFh6lw23pXHYNiKpjFGKoUKmwrOwHyS3lnFQ78O6WUjq-OM1ITxXTrgD6P5K2E89TnqOgGcaOyn6aLaVIcvPhARMyQEqlFDI2mcnYwIfwADKB-&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,olHwwJjC2TUJmGxt3e1EpgRMfVMAAdmudKrqZ5ZByilJc51VAK_ZZHGL1mLqUqv59mtQqrIVOT0FHxcuVGnfmTsHrtP_Lfh6vEjgZmLr0Aa8EqAKSU0BDQ,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,3crdMkhGhlkrJXp-HsjZIEJOjIoajfvvI_g-uTOHgEc26-rm5spEwaDR4tTrIOKeN5HSQz3HfKcnanjUMfuquaMfvk7M7K2AtlUVxZdR9Q,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,Qv7RvLuSjcsV2_72qHHLM6lmOI05RppEGGyR8280SVuKWQgOAl-EeFo8ZDeB_9Hw2f5ZgHbKDJbuB11rT-FBX6wf71ASJsIGNZYXzHn1HnNPy3pXaBaEgvrJvA,,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,7hPEKeOrrW0p5qUGMytIu7avFNFhxNhXFzKgo6a-hDqm4ky01dphgxAxTpF5DmP3oUrVV1uWYwMbJuUFv1IoX_o9-H7Y6xXWdj067c1A&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,dU7lk32qJcXeW-cUtM1E65BB47JiLhSgJYTqhLIX373Pj0miueYi3kN7MYtAU5zT39qOtzzAiHcBVYWsGDB3N8R31seI_xbm-rQzfd0jzb6V1rSLy1kotiI,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,BtR0iFYtXaxmAlygBDvC74BLAVELzJivsFTs_YS41zS_XagPPAr0UUKW1lNrafWE3WxaZcW_S5WbNn583_uHQ7aLAA9vskUJfxaMwUrM_DxAOd2N&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,nW_o_ayhGlEOytZThvhNsFjqKDq7oEaYAEjw_jVKEHAnXglYBlSGSA3yj7wyG-2DXRXiRZSD11UwsKx7fSGPuHkUUSoULIYqacVjSrfmUuEfG_A,&typo=1&ancr_add=1>)
2024-01-10T19:52:34.824 [Thread-19] JmsStaticConnectionPool.onException:78 [] ERROR - Receive error occurred. javax.jms.JMSException: ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:714<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a714&c=E,1,3FhMyZ8H2FihvFzilmw5MqmH_tkqp5vHbCOJsN5x6sQNSLyFELuJt_Ej2M_24CH42Xx0iQesALGDJK8HHdR_PDaLploHa8R84-uwfftA3DUM&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:735<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a735&c=E,1,JvpZWdXDc673FDsIBWz8WgGu29y6j8H0zAczvf_UXYyUAzEMKtvXJ3tMkleaFsrHU-QdlAUi0bWddjnikuurWEEMfZs0jIDWZ3jnrx2__Vbg93yZ8oo,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:868<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a868&c=E,1,AznnL1cSXDTbCj6iQ4ovEeP71-oopbCcH0laxV9dB-Y_kBuerNfeOAXRkRLybAqgqQBjJZKBNFEoveCJjTGncMCguHnY0hvsdZuKIHz6BEhvRX7N&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:856<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a856&c=E,1,vBewrTh4rfp_ezwl572jqeiy0VBShHwQJNRT99yv4B54VM5Qaf2hTHrOq33emXwk_WSFrcueJSCKI90AxcN4jUznfP1ZLzL1dJdUz01zK4vN&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:802<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a802&c=E,1,EY-ry2DzdJloXWtaemW-dWnRy1-rr2mQFFjoCbmWjX6xHCNZF95ldY2zUgfhOhSQFfzm7eYV0PdRUTOwJrx9jCD4U-B5ztFqWmvqEmJQuqdCyQgypA,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,gHouFFMo7plejV1zi4ureD9QEDkW-WjvytRlH6DOyEt7oMO8ULkj6APov-gDhxtXAspTlk7qF6RYw-RrCqfSrXlcE0Cm45J61qJHt5Yc3zuBJJyEX8A,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,17MGSyeFWTwZKhb2DCqWeyFvdlinlKAunc4JSHJltjjiaCeT_1UjXFsTMtEiF_NYIOMI3brJbPkRW1QkJIhe_Br44jy1od8-BI3l37IfFJzZbcculrc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,F6NyfiAiJI9fxHnOQnnpO7dLyoFavqwnNivvLL6x2LO54G8SpNhXjlUN748SkNWwe_oCD9JhI7u_BOk1uNbwSTYV0lXRyewBrwvvysSEOSCggiFCFRgjtA,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,TnFdrKl6nCZv9PR6lSsRZ2YN0r8LqpGQMRo3oHlgyE7_AgUMzrW12R4HeB5JhX8Akv-1iKeVGrnTLV2WeAbBOCM-fukhfRPsdUgq5pwe0yU,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,vAtPtyOuzaufOELKjMWnOBMrJEOShH0EzH9G3WL8gak4hWHmnVWPL1W7c_YnxRH2IerBQErg9VJApOiiX6TR6tcPau3xqRHrPyrsH70KHg2v&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,pak9EX0xrMkRTN71voUljZIOmUvksqdCCqWdwh8fq2R4OZXUQFIHNH3P9j0t-wpQStfSs_ZsT0a9BZYopaoyMSnMkN3xqvXn8A9dTehv&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,acJEflBFObRkFNBskLYk7AARfS1_CzZH3yFGYB9r_Ype7n0DAWduZhGpmImVc9k1pTnmFoA8L8UukMVvWBinxujU6r75ipmzh4I_cNQ5Gzmd97g5SAU4dRsf3Hw,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,aM5tVNMvOtLUxuoV8Dez1Y7fAS8KwME3peX3IqP_QPViIWgkZQahJW_1BMBQUo35u5h9hGJ3XSyDgkECwWOGDn6a9W-gctjh8WvUKx9yQ0sSLwheMcs5mhR2yVM,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,fUWcgLX3hupjGCjPYEByZ8xfzFx4iN-VnkLB9MarNu6OEOfrwnCkvJvtu6e-ZV-FREp0Ssisi_fezDu18eP9adS4mmcPPlJH69vnf0YDVg,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,KuAjbB9H9ddyi6unfXrVdNXylpB7l6qLIUfJARScyUnc7iH1GtvO0s1MAtB9fwkdqXq8idOJYZs_rg0iU-sN4aV24ID0Os5tsyvB34ba3ve2D3diROqixzkQuw4,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,EcYd8WT9GtMN9IXo58gsQpj5uNwcdCRtDfa0zG9t2jpvfdaSX5r9vLey2icJ-4wo3EyBMyUQzwyqYwC5BMfe4yHGjyo7h8jxmjcEKig4V0KyVa_SVq58NBT8&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,_a7OW7NorEoCm9eOnYjCi6TFl2__or_ickK-DwP4rrMmRX2Rj_TKaxpGjSAwCJZasllt1gefhShoob68MthMyp6bxv9N5fucG2HiF0g3yK4kqHTZjiFn7kxZ&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,-ttWHQzQDZ94E-HqKtYTKV2nuApkTk60Zy25iWFIykqfNe0o2seEul4UE_gETZr08B40bJjmZKMHq_-R0M8bQ8Fbu8twSkWx9MPLFUhc&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,624T4PuCyLmsS-UG0IoXRx3MMz4DqnU_1BuWfOz973WSpqfLsFFDAICu75KCoCwAs9O8WgKTEcBVqRhhf1JUqI3Znxem3LsG7Ml0Td3DnJJUoGYR0BE,&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,7UL50c_ahB0IhSHbe9OlGABBKKE4OFyNM67SjmNrSo_LQhfViHAlLgvwdlRpbmDdymI5VaeOEklHjvlweLBz8Ske6wjc8J3CzM05VYenh6DzNiI,&typo=1&ancr_add=1>)
Caused by: ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
... 7 more
2024-01-10T19:52:34.828 [Thread-17 (ActiveMQ-client-global-threads)] ClientSessionImpl.acknowledge:1225 [] DEBUG - client ack messageID = 10054445
2024-01-10T19:52:34.862 [Thread-24 (ActiveMQ-client-global-threads)] JmsServerBase.logRequest:117 [] INFO - REQUEST IN:
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,01YVdf0IHIl4w2xIwUh5i7ToRiQfv6TxT-FRWSNW0n6e2aucHkWVyKxkYJTpG1jt7_EUCuQVS0NWGalDyf11AUIb3qVPj67IUtjJB484IdLKrHgbmSd-Q8Vg7AKZ&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Tuesday, January 9, 2024 1:57 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Both the "replicated-failback-static" and "replicated-failback" examples run without issue out of the box on my machine so I'm not sure what's going on with that. One thing that's odd is that out-of-the-box the line 122 [1] doesn't match the method call in your NPE. Perhaps you modified the example?
Aside from that there were a handful of questions in my previous message you didn't address. I think the questions about thread dumps are the most pressing given that you indicated you're deleting the pod. My guess is that the client is getting hung in a network call that doesn't want to return due to the pod disappearing. A series of thread dumps during the minute delay would confirm or deny that.
Justin
[1] https://github.com/apache/activemq-artemis/blob/2.31.2/examples/features/ha/replicated-failback-static/src/main/java/org/apache/activemq/artemis/jms/example/ReplicatedFailbackStaticExample.java#L122
On Tue, Jan 9, 2024 at 12:47 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hey Justing, thanks for the help.
We are using version 2.31.2, which is pretty recent, if not the latest.
With regards to the log traces, other AMQ threads show similar behavior, and our system isn’t doing much until AMQ-client stabilizes.
I am running on a k3d cluster and the failover is triggered by me deleting the dm-activemq-live pod, which k8s then spins back up.
Also, our slave config does NOT specify “allow-failback=true”, so the spin up of the new dm-activemq-live instance does not trigger a fail-back, which is intentional to eliminate failover-failback chaos in such a short time.
I am trying to setup a simpler test case by using one of the examples, artemis/2.31.2/examples/features/ha/replicated-failback-static, however, I am getting this NPE on line 112:
Caused by: java.lang.NullPointerException: Cannot invoke "javax.jms.TextMessage.getText()" because "message0" is null
The NPE seems to indicate that, you cannot call consumer.receive() again even if NOT acknowledged, which seems to contradict the comments on the code. I have run into this same behavior in another example as well, artemis/2.31.2/examples/features/ha/replicated-failback.
Can you please confirm the expected behavior of the example code in the 2.31.2 version? I can then modify the code accordingly. Or please point me to a working sample code.
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,lIOPl76F2R4EtenIAEJMeQ17kfmHQ6buC7G1W5TRefOcSfxvj_-3dLXV3MKDkCTkac_65Rw0SYmof66i91zGwgb_SS1nKaBpao_Lw7JNNk1mkE33&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Monday, January 8, 2024 9:20 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> Now, the backup process does start processing msgs right away...
What evidence do you have of that?
> ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable.
The log snippets you provided show a reconnection attempt back to dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to dm-activemq-backup-svc:61617 both of which fail almost immediately. Then right after that another connection attempt is made to dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e. from 20:03:44.874 to 20:04:44.933). Are there any other logs during this minute? Did you happen to get any thread dumps during this time? If so, what did they show? If not, could you? What was happening with dm-activemq-live-svc during this time? Perhaps there was something weird going on with the network related to whatever you were doing with that machine to trigger failover.
You might consider simplifying your test-case and then adding complexity bit by bit to see what triggers the difference in behavior. For example, run your test with the client and both brokers on the *same machine*. If that works then move just the client to a remote machine. If that works then test the client just reconnecting to the primary broker upon restart (i.e. no failover). Then finally put all three entities on separate machines and test failover. Hopefully such an exercise will give you a better understanding of where the problem might lie.
Lastly, please indicate what version of ActiveMQ Artemis you're using.
Justin
On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
I’m working with John Lilley on getting an ActiveMQ HA configuration operational. Our current HA config is replication with master and slave instances and with static connections, and it seems to be working, except that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect itself after a failover from the live to the backup instance. Now, the backup process does start processing msgs right away, but the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable. I have included the relevant log traces from the Thread-0 ActiveMQ-client showing the Connection failure detection all the way through when the client is usable again, over 1 minute later. I am hoping that, based on the trace, you can find something that, perhaps, we should be doing differently in the client side?
Here is our Client connection url:
(tcp://dm-activemq-live-svc: 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
Should we also be including the corresponding connector names?
In an earlier email with John Lilley you recommended 2 brokers using “shared storage” instead of “replication”, which I will try next, however, since this issue seems to be on the client-side, I want to make sure our client-side is setup correctly.
Thanks in advance for your help.
Lino
Thread-0 log traces:
2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
---
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616>
2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617<http://10.43.11.76:61617>
2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
---
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616>
2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = ced14fbc
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617<http://10.43.11.76:61617>
2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 10, performing as a failover operation now and recreating objects
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,hqiYs6vnJzfq-X42G9W9ohJtYgn9LRPR-82RHxMyPnKUD5xBwCbo8W6b-PegfmAGwvEENlDO0k2IkoGFFegGxR-H6J72UHHlpEFbGtJzhCF3sucQ&typo=1&ancr_add=1>
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,KX9atZvl9XxpGPWobW9hXsmaLrVekfw2221zvD0PJ9Ik3fEQVyHKcZZCe7MQRciamxD-T-WDEk5Xa7_cAt0PSHwVW3jG7zzhq8XH9XIRC8EXmg,,&typo=1&ancr_add=1>, needed = 65536, arriving = 0
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,ZOCwWoQv34cYLuw9hZvJzBHfSrX6oQeQ_3-SetFMJa1hRC6lXgiKsAElOhfZ_vfnBhRGRJPZjSOqHIbwWO2ltSYYY7ElQjyItp0uDo79_jaDkw,,&typo=1&ancr_add=1>
2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,obbNTaFqn6mglgyZXy1S-0nBLNVtEYlkZBwAEKSGDGN0RuqlR_-7fIQPW4ELNqVFJdenAcKh65C3yQ2b_Lef7W01Tst6ZTrmI66KfHLae572jXv-HF3O&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ytVVTXzfq-J1NXl2ct7T_47F5EbfZNQTXP8dlqw720EiCe32c6CLwlpk0-v4MzUD61FErq0nkRTalFpcTtIN8TDeBvz70rzb__TyeIDm48ZKfFU28qQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,KQYUuEgOaR6V8a_ZC_xBpKZmDneB4LTL1cQzn2Gqtla8ntHJ1fgRs9KelLgvbMd9zvBTg8maXUerYZYQZa-DlfK7fon0JZ0O74PNJ1SOCHqIPN4fZQ6Xv731OGOD&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,OSVh6NSORxz7qZ6eIuUDgWskoYfyXT4NNdjJYGMbH7HeWEUiymys2DnptjYKdmQfqjfHmI-egRZ31yWwd9MKzEd-gbbsuTUsqn_46niKaw,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,DM2boEBGg44zzcd0m37ITkOnfDrkqLE8ZSFGJpEI4AGo2GyOI2yKX08AEkKHxw_UnDMF1vccYEvESFDXjXBBjjIGO5TEsMLWZHbJF3AOIEQ73uc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,wipgGeA7UVZIy1uuqvuxJEx4BA8TvYfSYmO617IgrvnOo-z7sn1vUjXFqIEU9M1HmedBVoHxB_LpvuEZ9p_biioPM8V-V9eS8i8oST46GvBhZZIc&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,0FNTjscT_d2CNIid0J8d_x7C7Peb6_neE-a_MZeKqkM1o_sPGujvpkXuiLv6kms9wSFyYqSux86C1MX2R8WKM8IZBsjKt7sCs03U2lRYlVbw2eTDg0ihSEV1nA,,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3yKPqxpIi9n43QJSCQiysKGlEYttZ5Dmuirc7jUAWNz6orLfIUCYwdDEJ3eU0QiHywsNAJdWqrvnx5rZK6-8TTJb6o8tr1WSvSL7omLF&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,6B4qLrj7q9DzDb0Trp9xgst7cOBhbD4hcEcGyPFul5E_mh3_oGmColB9YQkQTB1gJ6x5Mcm0VwvBLddpDpfFkcZwlzb4YNR9Ym5fcDclFW97pIrQ9oG4&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KqR93A_gEw1FpU7__PsYjSkOC5LH0zFTSKu5K9YlCdVAz3h_c9EkBndO7aliE4RSbK_qslEZp0b-YzJfCOxmB22jAfplsYxguMl1qjC0rWpNjfr9_fh4RjNL3AOK&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,mhChlKkihYjDw_Kk5SL1d_4eb7zhBlSzri_nwjOUsrs6MAvEfDAAjia07yTMUU6CQqc0WbDS2Fn8HJJI0UwJh1N7SKiz9dsP1gJLmgY66huLhXZE9c-3ECGXyIWQ&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,nn8IDmg6XQJLLXzH7kunXDNZa_3A7fUlnHdRCLoha7gQWJO1QeY32duGFAQMYaFHsfIbHbv142mei8HfCqRwrpuUfikYxLkFoXaD3pORZb9roBJs0H8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,DPLEvt-MPmmzv78LEkeXtYHM2q37m12TxX-nqjuIJurQdeXydsSakco9t_O7jgBbQ6y115xxdDb8IonFZbM86zgRDLcvUmeOSJuDmZybD_6JaTX0zpbQ&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,1vNFV-mDaLK92lJo0Mq6jvkCDzuwrSHT4lp1nB6KTzOCSO6comH5xNpEcvtZ_9pc0cQCD2kgDfB5xdxg0tAX2dF5oG5DHY6kxWXOBLnuAwiz1BkEFhw,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,aLydz0SGsclotqCro2upcTg44B_Ij7ckjApLu4B5JUwZKKdMo5z3RyUR4ikHFY8L3t_yB87Zp5atGRwoAPA1L9Mc9c681UjFrG16n-udVjeHtxQNxx8V-Nug&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1mZX-u-XHEZ_e4Ut7YETRvWm7U_qkC-q9QaPeHah4K4yNl5foyM1VSExCtYiisknkelDj6_K2MUHGc65Kd-6-JG_TXLby0OPWodJVjX3BtKaDNBUzusgpzT6ajL7&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-nbEPDxI7J-DFvbJfVVt2Cs1Vi_hN5bw-8vQF3SlM1nthw3VbJLWmS1h73cupUxqBoHCImTzDqI1m5gutx3ZriLpLpU7IqutHCqZuT6CTVWACPtj&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,-lUcdhDzaHWdAWLCc0IxWCFkKtKHxoEd64GWhzXAbeuG_Zm9mvn624BY6uS8MgQn-s2TMF8PbEPYVbOOy6BJuI8ByeiuAF9Y8f2I5wib&typo=1&ancr_add=1>)
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,T2UV-1dji1RODRcU4Ff4j_IzfTH2n47BatzyLZcEOU8VROFr3GS8re7CjDcnnHCSaPEtR-35J3z30dnYcz5K66rVsMVxgDiyh0FvEDKnN67naHjYbn0NpzAc&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
Posted by Justin Bertram <jb...@apache.org>.
> So our issue seems to be only for failover from live-to-backup,
consistently.
As far as I know the client doesn't make any distinction between these two
cases so I'm guessing there's some kind of environmental difference that's
making an impact.
> Is there something that we are perhaps not doing on the client?
I don't think so. The connection URL you provided previously looks good.
> I will extend one of the example codes to use threads and sone of the
AMQ-client wrapper code in an effort to isolate this behavior on a simpler
code base, as you recommented.
Before going down that road it might be better to inspect the ostensibly
low-hanging fruit first. Did you happen to test everything locally? If so,
what happened? Did you acquire any thread dumps during the delay as
previously requested?
Justin
On Wed, Jan 10, 2024 at 3:04 PM Lino Pereira
<li...@redpointglobal.com.invalid> wrote:
> Hey Justin,
>
>
>
> After starting fresh with version 2.31.2, the examples work out of the box
> for me as well.
>
> And, after ensuring that version 2.31.2 is running everywhere, client was
> still using 2.30.0, and refactoring to using the recommended shared-store
> HA, our AMQ-client is still behaving as before. That is, failover from
> live-to-backup, client recovery takes over 1 minute, and logs report what I
> posted earlier. However, failover from backup-to-live, client recovers
> within 5 seconds, which is the expected behavior, and our system is happy.
> Please see new log excerpts with more logging included for the 2 relevant
> threads.
>
>
>
> So our issue seems to be only for failover from live-to-backup,
> consistently. Is there something that we are perhaps not doing on the
> client?
>
>
>
> I will extend one of the example codes to use threads and sone of the
> AMQ-client wrapper code in an effort to isolate this behavior on a simpler
> code base, as you recommented.
>
>
>
> In the meantime, I thought I’d ping you in case you can think of something
> obvious that we might have missed, especially since we just moved from
> 2.30.0.
>
>
>
> Thanks again,
>
> Lino
>
>
>
> Our versions:
>
> ARTEMIS_VERSION=2.31.2
>
> JMX_VERSION=0.18.0
>
>
>
> Logs from failover from backup-to-live:
>
> 2024-01-10T19:52:29.588 [Thread-24 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-backup-svc/10.43.229.161:61617 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d7ff7885, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1172)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
>
>
> 2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-backup-svc/10.43.229.161:61617 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d43c72b6, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1172)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
>
>
> 2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:29.592 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:29.594 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:29.593 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:29.597 [Thread-6 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d4cf19d5
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.600 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 7b2392ac
>
> 2024-01-10T19:52:29.602 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.602 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.603 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = c1fb4c82
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.610 [Thread-8 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 06b7b3a3
>
> 2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:29.611 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 65680be4
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.624 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 97a109b7
>
> 2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:29.625 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:29.629 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 77b2de2f
>
> 2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
>
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:30.612 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = f92d7cd6
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.613 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 1d44895d
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.614 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = e8602acd
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:30.631 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 0a217bfd
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.632 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d2aa00d7
>
> 2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.633 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 9a4f639e
>
> 2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:30.634 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = ecdb1cbf
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:31.000 [Schedule-Runner]
> SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service
> as system
>
> 2024-01-10T19:52:32.001 [Schedule-Runner]
> SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service
> as system
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 2/200
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:32.116 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = a6b3d488
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.117 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = c0627cd9
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.118 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 71523eb3
>
> 2024-01-10T19:52:32.119 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.120 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> 2024-01-10T19:52:32.134 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 2/200
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:32.135 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 68efd0dc
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.137 [Thread-1 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = daddef73
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and previousConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.137 [Thread-3 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 84813ce3
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and initialConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:32.138 [Thread-5 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 533990e9
>
> 2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
>
>
> 2024-01-10T19:52:34.370 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 3/200
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:34.371 [Thread-7 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 1d8df71b
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:34.373 [Thread-9 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = de20e193
>
> 2024-01-10T19:52:34.373 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 3/200
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and currentConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-10T19:52:34.390 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.229.161:61617
>
> 2024-01-10T19:52:34.390 [Thread-10 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = 4661fd5d
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.45.150:61616
>
> 2024-01-10T19:52:34.392 [Thread-11 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = d1c14a74
>
> 2024-01-10T19:52:34.392 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.404 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-10T19:52:34.405 [Thread-9 (ActiveMQ-client-netty-threads)]
> ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up
> backup config = TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc for live =
> TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.405 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 38, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.409 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-10T19:52:34.410 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 37, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.411 [Thread-24 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 10, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.411 [Thread-11 (ActiveMQ-client-netty-threads)]
> ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up
> backup config = TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc for live =
> TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc
>
> 2024-01-10T19:52:34.413 [Thread-24 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> activemq.management
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address activemq.management, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address activemq.management
>
> 2024-01-10T19:52:34.415 [Thread-24 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d7ff7885, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410)
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407)
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98)
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:212)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1172)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
>
>
> 2024-01-10T19:52:34.435 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 54, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.465 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 98, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.474 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 20, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.483 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 128, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.495 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 158, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.499 [Thread-4]
> AbstractProducerCreditsImpl.checkCredits:184 [] DEBUG - not asking for -1
> credits on activemq.management
>
> 2024-01-10T19:52:34.499 [Thread-4]
> AbstractProducerCreditsImpl.afterAcquired:98 [] DEBUG - AfterAcquire 333
> credits on address activemq.management, pendingCredits=65203
>
> 2024-01-10T19:52:34.506 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 106, performing as a failover operation now and recreating
> objects
>
>
>
> 2024-01-10T19:52:34.680 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 155, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.683 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 21, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.684 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 129, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.686 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 57, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.687 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 174, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.688 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 176, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.689 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 56, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.690 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 108, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.691 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 156, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 233, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 112, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.693 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 187, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.694 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 110, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.695 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 144, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.696 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 235, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.697 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 28, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 47, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 30, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.699 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 77, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.701 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 65, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.710 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 70, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 34, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 159, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.713 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 76, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.714 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 119, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.716 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 59, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.719 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 87, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.720 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 80, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 166, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 149, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.722 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 137, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.723 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 163, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.724 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 179, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.726 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 181, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 17, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 58, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.728 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 32, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 49, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 234, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.730 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 143, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.731 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 42, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 136, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 178, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.733 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 55, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.734 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 95, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.737 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 48, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 16, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 146, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.739 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 22, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.742 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 43, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.743 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 130, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.744 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 36, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.745 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 60, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.746 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 117, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.751 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 14, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.752 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 81, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.754 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 93, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 138, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 100, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.756 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 62, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.757 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 125, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.758 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 15, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 99, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 44, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.760 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 172, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.761 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 84, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 79, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 122, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.764 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 64, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.765 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 88, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.767 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 151, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.772 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.773 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 96, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.774 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 27, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.775 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 73, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.776 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 90, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.777 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 134, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.778 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 35, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.780 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 118, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 186, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 41, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.782 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 31, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 150, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 145, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.787 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 127, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.789 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 132, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 170, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 142, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.791 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 53, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.792 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 86, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.793 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 147, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 115, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 12, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.795 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 135, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.796 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 68, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.797 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 103, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.798 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 141, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 165, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 18, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.800 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 33, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.801 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 116, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.802 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 124, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 26, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 50, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.804 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 75, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 85, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 69, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 97, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 101, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.808 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 153, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.809 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 107, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.810 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 175, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.811 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 66, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.812 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 111, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 152, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 45, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.814 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 61, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.815 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 113, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.816 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 105, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 162, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 171, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.818 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 173, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 184, performing as a failover operation now and recreating
> objects
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.820 [Thread-15 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.afterAcquired:98 [aoy5vnfnjetz,6ek4wlg8c0ad]
> DEBUG - AfterAcquire 781 credits on address rpdm.v1.authentication,
> pendingCredits=63974
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.start.project.execute.2048.250
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.user.group.query
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.user.group.query, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.user.group.query
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c27986b0-f570-4599-b651-7153b4f0fa40
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.cache.invalidation.topic, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.cache.invalidation.topic
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.authentication
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.authentication
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.version.migration
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.message.logs
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.canvas.state.query
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.canvas.state.query, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.canvas.state.query
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> rpdm.v1.permissions
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address rpdm.v1.permissions
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address
> fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536,
> arriving = 0
>
> 2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536
> credits on address fe52afa6-a998-4101-b382-495d8e48cd52
>
> 2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:184 [z7wx0g93sdr7] DEBUG - not
> asking for -1 credits on acf62840-864a-4eb4-a902-4861db94eb93
>
> 2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.afterAcquired:98 [z7wx0g93sdr7] DEBUG -
> AfterAcquire 505 credits on address acf62840-864a-4eb4-a902-4861db94eb93,
> pendingCredits=65031
>
> 2024-01-10T19:52:34.824 [Thread-26 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=d43c72b6, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410)
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407)
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98)
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:212)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1172)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
>
>
> 2024-01-10T19:52:34.824 [Thread-19] JmsStaticConnectionPool.onException:78
> [] ERROR - Receive error occurred. javax.jms.JMSException:
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:714)
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:735)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:868)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:856)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:802)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407)
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98)
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:212)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1172)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
> Caused by: ActiveMQDisconnectedException[errorType=DISCONNECTED
> message=AMQ219015: The connection was disconnected because of server
> shutdown]
>
> ... 7 more
>
>
>
> 2024-01-10T19:52:34.828 [Thread-17 (ActiveMQ-client-global-threads)]
> ClientSessionImpl.acknowledge:1225 [] DEBUG - client ack messageID =
> 10054445
>
>
>
> 2024-01-10T19:52:34.862 [Thread-24 (ActiveMQ-client-global-threads)]
> JmsServerBase.logRequest:117 [] INFO - REQUEST IN:
>
>
>
>
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> Lino Pereira
>
> C++ Developer, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Tuesday, January 9, 2024 1:57 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Both the "replicated-failback-static" and "replicated-failback" examples
> run without issue out of the box on my machine so I'm not sure what's going
> on with that. One thing that's odd is that out-of-the-box the line 122 [1]
> doesn't match the method call in your NPE. Perhaps you modified the example?
>
>
>
> Aside from that there were a handful of questions in my previous message
> you didn't address. I think the questions about thread dumps are the most
> pressing given that you indicated you're deleting the pod. My guess is that
> the client is getting hung in a network call that doesn't want to return
> due to the pod disappearing. A series of thread dumps during the minute
> delay would confirm or deny that.
>
>
>
>
>
> Justin
>
>
>
> [1]
> https://github.com/apache/activemq-artemis/blob/2.31.2/examples/features/ha/replicated-failback-static/src/main/java/org/apache/activemq/artemis/jms/example/ReplicatedFailbackStaticExample.java#L122
>
>
>
> On Tue, Jan 9, 2024 at 12:47 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hey Justing, thanks for the help.
>
>
>
> We are using version 2.31.2, which is pretty recent, if not the latest.
>
>
>
> With regards to the log traces, other AMQ threads show similar behavior,
> and our system isn’t doing much until AMQ-client stabilizes.
>
> I am running on a k3d cluster and the failover is triggered by me deleting
> the dm-activemq-live pod, which k8s then spins back up.
>
> Also, our slave config does NOT specify “allow-failback=true”, so the spin
> up of the new dm-activemq-live instance does not trigger a fail-back, which
> is intentional to eliminate failover-failback chaos in such a short time.
>
>
>
> I am trying to setup a simpler test case by using one of the examples,
> artemis/2.31.2/examples/features/ha/replicated-failback-static, however, I
> am getting this NPE on line 112:
>
> Caused by: java.lang.NullPointerException: Cannot invoke
> "javax.jms.TextMessage.getText()" because "message0" is null
>
>
>
> The NPE seems to indicate that, you cannot call consumer.receive() again
> even if NOT acknowledged, which seems to contradict the comments on the
> code. I have run into this same behavior in another example as well,
> artemis/2.31.2/examples/features/ha/replicated-failback.
>
>
>
> Can you please confirm the expected behavior of the example code in the
> 2.31.2 version? I can then modify the code accordingly. Or please point me
> to a working sample code.
>
>
>
> Thanks,
>
> Lino
>
>
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,lIOPl76F2R4EtenIAEJMeQ17kfmHQ6buC7G1W5TRefOcSfxvj_-3dLXV3MKDkCTkac_65Rw0SYmof66i91zGwgb_SS1nKaBpao_Lw7JNNk1mkE33&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Monday, January 8, 2024 9:20 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > Now, the backup process does start processing msgs right away...
>
>
>
> What evidence do you have of that?
>
>
>
> > ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute
> apart, before it stabilizes after the failover and is once again usable.
>
>
>
> The log snippets you provided show a reconnection attempt back to
> dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to
> dm-activemq-backup-svc:61617 both of which fail almost immediately. Then
> right after that another connection attempt is made to
> dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e.
> from 20:03:44.874 to 20:04:44.933). Are there any other logs during this
> minute? Did you happen to get any thread dumps during this time? If so,
> what did they show? If not, could you? What was happening with
> dm-activemq-live-svc during this time? Perhaps there was something weird
> going on with the network related to whatever you were doing with that
> machine to trigger failover.
>
>
>
> You might consider simplifying your test-case and then adding complexity
> bit by bit to see what triggers the difference in behavior. For example,
> run your test with the client and both brokers on the *same machine*. If
> that works then move just the client to a remote machine. If that works
> then test the client just reconnecting to the primary broker upon restart
> (i.e. no failover). Then finally put all three entities on separate
> machines and test failover. Hopefully such an exercise will give you a
> better understanding of where the problem might lie.
>
>
>
> Lastly, please indicate what version of ActiveMQ Artemis you're using.
>
>
>
>
>
> Justin
>
>
>
> On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hi Justin,
>
> I’m working with John Lilley on getting an ActiveMQ HA configuration
> operational. Our current HA config is replication with master and slave
> instances and with static connections, and it seems to be working, except
> that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect
> itself after a failover from the live to the backup instance. Now, the
> backup process does start processing msgs right away, but the
> ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart,
> before it stabilizes after the failover and is once again usable. I have
> included the relevant log traces from the Thread-0 ActiveMQ-client showing
> the Connection failure detection all the way through when the client is
> usable again, over 1 minute later. I am hoping that, based on the trace,
> you can find something that, perhaps, we should be doing differently in the
> client side?
>
>
>
> Here is our Client connection url:
>
> (tcp://dm-activemq-live-svc:
> 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
>
>
>
> Should we also be including the corresponding connector names?
>
>
>
> In an earlier email with John Lilley you recommended 2 brokers using
> “shared storage” instead of “replication”, which I will try next, however,
> since this issue seems to be on the client-side, I want to make sure our
> client-side is setup correctly.
>
>
>
> Thanks in advance for your help.
>
> Lino
>
>
>
> Thread-0 log traces:
>
> 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-live-svc/10.43.197.248:61616 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=a8a4a77c, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> ---
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and currentConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.197.248:61616
>
>
>
> 2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.11.76:61617
>
>
>
> 2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> ---
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and currentConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.197.248:61616
>
> 2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = ced14fbc
>
>
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.11.76:61617
>
>
>
> 2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
>
>
> 2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 10, performing as a failover operation now and recreating
> objects
>
>
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address
> activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,hqiYs6vnJzfq-X42G9W9ohJtYgn9LRPR-82RHxMyPnKUD5xBwCbo8W6b-PegfmAGwvEENlDO0k2IkoGFFegGxR-H6J72UHHlpEFbGtJzhCF3sucQ&typo=1&ancr_add=1>
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,KX9atZvl9XxpGPWobW9hXsmaLrVekfw2221zvD0PJ9Ik3fEQVyHKcZZCe7MQRciamxD-T-WDEk5Xa7_cAt0PSHwVW3jG7zzhq8XH9XIRC8EXmg,,&typo=1&ancr_add=1>,
> needed = 65536, arriving = 0
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,ZOCwWoQv34cYLuw9hZvJzBHfSrX6oQeQ_3-SetFMJa1hRC6lXgiKsAElOhfZ_vfnBhRGRJPZjSOqHIbwWO2ltSYYY7ElQjyItp0uDo79_jaDkw,,&typo=1&ancr_add=1>
>
> 2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=a8a4a77c, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> ClientSessionFactoryImpl.java:410
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,obbNTaFqn6mglgyZXy1S-0nBLNVtEYlkZBwAEKSGDGN0RuqlR_-7fIQPW4ELNqVFJdenAcKh65C3yQ2b_Lef7W01Tst6ZTrmI66KfHLae572jXv-HF3O&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> NettyConnector.java:835
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ytVVTXzfq-J1NXl2ct7T_47F5EbfZNQTXP8dlqw720EiCe32c6CLwlpk0-v4MzUD61FErq0nkRTalFpcTtIN8TDeBvz70rzb__TyeIDm48ZKfFU28qQ,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:666
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,KQYUuEgOaR6V8a_ZC_xBpKZmDneB4LTL1cQzn2Gqtla8ntHJ1fgRs9KelLgvbMd9zvBTg8maXUerYZYQZa-DlfK7fon0JZ0O74PNJ1SOCHqIPN4fZQ6Xv731OGOD&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,OSVh6NSORxz7qZ6eIuUDgWskoYfyXT4NNdjJYGMbH7HeWEUiymys2DnptjYKdmQfqjfHmI-egRZ31yWwd9MKzEd-gbbsuTUsqn_46niKaw,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,DM2boEBGg44zzcd0m37ITkOnfDrkqLE8ZSFGJpEI4AGo2GyOI2yKX08AEkKHxw_UnDMF1vccYEvESFDXjXBBjjIGO5TEsMLWZHbJF3AOIEQ73uc,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,wipgGeA7UVZIy1uuqvuxJEx4BA8TvYfSYmO617IgrvnOo-z7sn1vUjXFqIEU9M1HmedBVoHxB_LpvuEZ9p_biioPM8V-V9eS8i8oST46GvBhZZIc&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,0FNTjscT_d2CNIid0J8d_x7C7Peb6_neE-a_MZeKqkM1o_sPGujvpkXuiLv6kms9wSFyYqSux86C1MX2R8WKM8IZBsjKt7sCs03U2lRYlVbw2eTDg0ihSEV1nA,,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3yKPqxpIi9n43QJSCQiysKGlEYttZ5Dmuirc7jUAWNz6orLfIUCYwdDEJ3eU0QiHywsNAJdWqrvnx5rZK6-8TTJb6o8tr1WSvSL7omLF&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,6B4qLrj7q9DzDb0Trp9xgst7cOBhbD4hcEcGyPFul5E_mh3_oGmColB9YQkQTB1gJ6x5Mcm0VwvBLddpDpfFkcZwlzb4YNR9Ym5fcDclFW97pIrQ9oG4&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KqR93A_gEw1FpU7__PsYjSkOC5LH0zFTSKu5K9YlCdVAz3h_c9EkBndO7aliE4RSbK_qslEZp0b-YzJfCOxmB22jAfplsYxguMl1qjC0rWpNjfr9_fh4RjNL3AOK&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,mhChlKkihYjDw_Kk5SL1d_4eb7zhBlSzri_nwjOUsrs6MAvEfDAAjia07yTMUU6CQqc0WbDS2Fn8HJJI0UwJh1N7SKiz9dsP1gJLmgY66huLhXZE9c-3ECGXyIWQ&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,nn8IDmg6XQJLLXzH7kunXDNZa_3A7fUlnHdRCLoha7gQWJO1QeY32duGFAQMYaFHsfIbHbv142mei8HfCqRwrpuUfikYxLkFoXaD3pORZb9roBJs0H8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,DPLEvt-MPmmzv78LEkeXtYHM2q37m12TxX-nqjuIJurQdeXydsSakco9t_O7jgBbQ6y115xxdDb8IonFZbM86zgRDLcvUmeOSJuDmZybD_6JaTX0zpbQ&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,1vNFV-mDaLK92lJo0Mq6jvkCDzuwrSHT4lp1nB6KTzOCSO6comH5xNpEcvtZ_9pc0cQCD2kgDfB5xdxg0tAX2dF5oG5DHY6kxWXOBLnuAwiz1BkEFhw,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,aLydz0SGsclotqCro2upcTg44B_Ij7ckjApLu4B5JUwZKKdMo5z3RyUR4ikHFY8L3t_yB87Zp5atGRwoAPA1L9Mc9c681UjFrG16n-udVjeHtxQNxx8V-Nug&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1mZX-u-XHEZ_e4Ut7YETRvWm7U_qkC-q9QaPeHah4K4yNl5foyM1VSExCtYiisknkelDj6_K2MUHGc65Kd-6-JG_TXLby0OPWodJVjX3BtKaDNBUzusgpzT6ajL7&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-nbEPDxI7J-DFvbJfVVt2Cs1Vi_hN5bw-8vQF3SlM1nthw3VbJLWmS1h73cupUxqBoHCImTzDqI1m5gutx3ZriLpLpU7IqutHCqZuT6CTVWACPtj&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,-lUcdhDzaHWdAWLCc0IxWCFkKtKHxoEd64GWhzXAbeuG_Zm9mvn624BY6uS8MgQn-s2TMF8PbEPYVbOOy6BJuI8ByeiuAF9Y8f2I5wib&typo=1&ancr_add=1>
> )
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,T2UV-1dji1RODRcU4Ff4j_IzfTH2n47BatzyLZcEOU8VROFr3GS8re7CjDcnnHCSaPEtR-35J3z30dnYcz5K66rVsMVxgDiyh0FvEDKnN67naHjYbn0NpzAc&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
RE: Looking for HA/replication boilerplate broker.xml and advice (continued)
Posted by Lino Pereira <li...@redpointglobal.com.INVALID>.
Hey Justin,
After starting fresh with version 2.31.2, the examples work out of the box for me as well.
And, after ensuring that version 2.31.2 is running everywhere, client was still using 2.30.0, and refactoring to using the recommended shared-store HA, our AMQ-client is still behaving as before. That is, failover from live-to-backup, client recovery takes over 1 minute, and logs report what I posted earlier. However, failover from backup-to-live, client recovers within 5 seconds, which is the expected behavior, and our system is happy. Please see new log excerpts with more logging included for the 2 relevant threads.
So our issue seems to be only for failover from live-to-backup, consistently. Is there something that we are perhaps not doing on the client?
I will extend one of the example codes to use threads and sone of the AMQ-client wrapper code in an effort to isolate this behavior on a simpler code base, as you recommented.
In the meantime, I thought I’d ping you in case you can think of something obvious that we might have missed, especially since we just moved from 2.30.0.
Thanks again,
Lino
Our versions:
ARTEMIS_VERSION=2.31.2
JMX_VERSION=0.18.0
Logs from failover from backup-to-live:
2024-01-10T19:52:29.588 [Thread-24 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-backup-svc/10.43.229.161:61617 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d7ff7885, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1172)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
2024-01-10T19:52:29.589 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-backup-svc/10.43.229.161:61617 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-10T19:52:29.590 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d43c72b6, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1172)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-10T19:52:29.592 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.593 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:29.592 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:29.594 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617
2024-01-10T19:52:29.593 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:29.596 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617
2024-01-10T19:52:29.597 [Thread-6 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d4cf19d5
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.597 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.600 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 7b2392ac
2024-01-10T19:52:29.602 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:29.602 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.603 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:29.603 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = c1fb4c82
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.604 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:29.610 [Thread-8 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 06b7b3a3
2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.610 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:29.611 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 65680be4
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.623 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:29.624 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 97a109b7
2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.624 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:29.625 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:29.628 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:29.629 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 77b2de2f
2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:29.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:30.611 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617
2024-01-10T19:52:30.612 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = f92d7cd6
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:30.612 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.613 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:30.613 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 1d44895d
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:30.614 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = e8602acd
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.614 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:30.630 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:30.631 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617
2024-01-10T19:52:30.631 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 0a217bfd
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:30.632 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d2aa00d7
2024-01-10T19:52:30.632 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:30.633 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 9a4f639e
2024-01-10T19:52:30.633 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:30.634 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = ecdb1cbf
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:30.634 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:31.000 [Schedule-Runner] SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service as system
2024-01-10T19:52:32.001 [Schedule-Runner] SystemAuthenticatorImpl.getAuthToken:67 [] DEBUG - Authenticating service as system
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 2/200
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:32.115 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617
2024-01-10T19:52:32.116 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = a6b3d488
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.117 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:32.117 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = c0627cd9
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.118 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:32.118 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 71523eb3
2024-01-10T19:52:32.119 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.120 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:32.134 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 2/200
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:32.135 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617
2024-01-10T19:52:32.135 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 68efd0dc
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.136 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:32.137 [Thread-1 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = daddef73
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and previousConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.137 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:32.137 [Thread-3 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 84813ce3
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and initialConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:32.138 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:32.138 [Thread-5 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 533990e9
2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:32.139 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
2024-01-10T19:52:34.370 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 3/200
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:34.371 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617
2024-01-10T19:52:34.371 [Thread-7 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 1d8df71b
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@daf22f0<ma...@daf22f0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:34.372 [Thread-26 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:34.373 [Thread-9 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = de20e193
2024-01-10T19:52:34.373 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 3/200
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and currentConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.389 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-10T19:52:34.390 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.229.161:61617
2024-01-10T19:52:34.390 [Thread-10 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = 4661fd5d
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@49f2ec37<ma...@49f2ec37> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-10T19:52:34.391 [Thread-24 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.45.150:61616
2024-01-10T19:52:34.392 [Thread-11 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = d1c14a74
2024-01-10T19:52:34.392 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.404 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-10T19:52:34.405 [Thread-9 (ActiveMQ-client-netty-threads)] ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up backup config = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc for live = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.405 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 38, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.409 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-10T19:52:34.410 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 37, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.411 [Thread-24 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 10, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.411 [Thread-11 (ActiveMQ-client-netty-threads)] ClientSessionFactoryImpl.setBackupConnector:317 [] DEBUG - Setting up backup config = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc for live = TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc
2024-01-10T19:52:34.413 [Thread-24 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address activemq.management
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address activemq.management, needed = 65536, arriving = 0
2024-01-10T19:52:34.414 [Thread-24 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address activemq.management
2024-01-10T19:52:34.415 [Thread-24 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d7ff7885, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:212)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1172)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
2024-01-10T19:52:34.435 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 54, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.465 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 98, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.474 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 20, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.483 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 128, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.495 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 158, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.499 [Thread-4] AbstractProducerCreditsImpl.checkCredits:184 [] DEBUG - not asking for -1 credits on activemq.management
2024-01-10T19:52:34.499 [Thread-4] AbstractProducerCreditsImpl.afterAcquired:98 [] DEBUG - AfterAcquire 333 credits on address activemq.management, pendingCredits=65203
2024-01-10T19:52:34.506 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 106, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.680 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 155, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.683 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 21, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.684 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 129, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.686 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 57, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.687 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 174, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.688 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 176, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.689 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 56, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.690 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 108, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.691 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 156, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 233, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.692 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 112, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.693 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 187, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.694 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 110, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.695 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 144, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.696 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 235, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.697 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 28, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 47, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.698 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 30, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.699 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 77, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.701 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 65, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.710 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 70, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 34, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.712 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 159, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.713 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 76, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.714 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 119, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.716 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 59, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.719 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 87, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.720 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 80, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 166, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.721 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 149, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.722 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 137, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.723 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 163, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.724 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 179, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.726 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 181, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 17, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.727 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 58, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.728 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 32, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 49, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.729 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 234, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.730 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 143, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.731 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 42, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 136, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.732 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 178, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.733 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 55, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.734 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 95, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.737 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 48, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 16, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.738 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 146, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.739 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 22, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.742 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 43, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.743 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 130, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.744 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 36, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.745 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 60, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.746 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 117, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.751 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 14, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.752 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 81, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.754 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 93, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 138, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.755 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 100, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.756 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 62, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.757 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 125, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.758 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 15, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 99, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.759 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 44, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.760 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 172, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.761 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 84, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 79, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.762 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 122, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.764 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 64, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.765 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 88, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.767 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 151, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.772 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.773 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 96, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.774 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 27, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.775 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 73, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.776 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 90, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.777 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 134, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.778 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 35, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.780 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 118, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 186, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.781 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 41, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.782 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 31, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 150, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.786 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 145, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.787 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 127, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.789 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 132, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 170, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.790 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 142, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.791 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 53, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.792 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 86, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.793 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 147, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 115, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.794 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 12, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.795 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 135, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.796 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 68, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.797 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 103, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.798 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 141, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 165, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.799 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 18, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.800 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 33, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.801 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 116, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.802 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 124, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 26, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.803 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 50, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.804 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 75, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 85, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.805 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 69, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 97, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.807 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 101, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.808 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 153, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.809 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 107, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.810 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 175, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.811 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 66, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.812 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 111, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 152, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.813 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 45, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.814 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 61, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.815 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 113, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.816 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 105, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 162, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.817 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 171, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.818 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 173, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 184, performing as a failover operation now and recreating objects
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.819 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.820 [Thread-15 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.afterAcquired:98 [aoy5vnfnjetz,6ek4wlg8c0ad] DEBUG - AfterAcquire 781 credits on address rpdm.v1.authentication, pendingCredits=63974
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.820 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.start.project.execute.2048.250, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.start.project.execute.2048.250
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.user.group.query
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.user.group.query, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.user.group.query
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.821 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c00b7a09-540c-4b59-a7af-99ef76d77e57
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c27986b0-f570-4599-b651-7153b4f0fa40
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.cache.invalidation.topic, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.cache.invalidation.topic
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 6c945e66-a227-4e1e-999f-2ab339d9c812
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 185a677f-f344-4973-9d3a-e01eeb102d30
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address b2d50731-8cd2-48a8-ac25-b541eae13f55
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address c3172442-182b-43fb-bb62-f47e9cdbd340
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.authentication
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.authentication, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.authentication
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.version.migration, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.version.migration
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.message.logs, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.message.logs
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.canvas.state.query
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.canvas.state.query, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.canvas.state.query
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address d6e3cc7f-ce14-4417-9453-6f29f1147b9b
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address rpdm.v1.permissions
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address rpdm.v1.permissions, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address rpdm.v1.permissions
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address 20f25377-43c3-4f8c-a40d-c8400c63589c
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:127 [] DEBUG - reset credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:180 [] DEBUG - Requesting 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52, needed = 65536, arriving = 0
2024-01-10T19:52:34.822 [Thread-26 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:189 [] DEBUG - Request 65536 credits on address fe52afa6-a998-4101-b382-495d8e48cd52
2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:184 [z7wx0g93sdr7] DEBUG - not asking for -1 credits on acf62840-864a-4eb4-a902-4861db94eb93
2024-01-10T19:52:34.824 [Thread-17 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.afterAcquired:98 [z7wx0g93sdr7] DEBUG - AfterAcquire 505 credits on address acf62840-864a-4eb4-a902-4861db94eb93, pendingCredits=65031
2024-01-10T19:52:34.824 [Thread-26 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=d43c72b6, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:212)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1172)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
2024-01-10T19:52:34.824 [Thread-19] JmsStaticConnectionPool.onException:78 [] ERROR - Receive error occurred. javax.jms.JMSException: ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:714)
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:735)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:868)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:856)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:802)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:212)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1172)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
... 7 more
2024-01-10T19:52:34.828 [Thread-17 (ActiveMQ-client-global-threads)] ClientSessionImpl.acknowledge:1225 [] DEBUG - client ack messageID = 10054445
2024-01-10T19:52:34.862 [Thread-24 (ActiveMQ-client-global-threads)] JmsServerBase.logRequest:117 [] INFO - REQUEST IN:
[rg] <https://www.redpointglobal.com/>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>
Sent: Tuesday, January 9, 2024 1:57 PM
To: users@activemq.apache.org
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Both the "replicated-failback-static" and "replicated-failback" examples run without issue out of the box on my machine so I'm not sure what's going on with that. One thing that's odd is that out-of-the-box the line 122 [1] doesn't match the method call in your NPE. Perhaps you modified the example?
Aside from that there were a handful of questions in my previous message you didn't address. I think the questions about thread dumps are the most pressing given that you indicated you're deleting the pod. My guess is that the client is getting hung in a network call that doesn't want to return due to the pod disappearing. A series of thread dumps during the minute delay would confirm or deny that.
Justin
[1] https://github.com/apache/activemq-artemis/blob/2.31.2/examples/features/ha/replicated-failback-static/src/main/java/org/apache/activemq/artemis/jms/example/ReplicatedFailbackStaticExample.java#L122
On Tue, Jan 9, 2024 at 12:47 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hey Justing, thanks for the help.
We are using version 2.31.2, which is pretty recent, if not the latest.
With regards to the log traces, other AMQ threads show similar behavior, and our system isn’t doing much until AMQ-client stabilizes.
I am running on a k3d cluster and the failover is triggered by me deleting the dm-activemq-live pod, which k8s then spins back up.
Also, our slave config does NOT specify “allow-failback=true”, so the spin up of the new dm-activemq-live instance does not trigger a fail-back, which is intentional to eliminate failover-failback chaos in such a short time.
I am trying to setup a simpler test case by using one of the examples, artemis/2.31.2/examples/features/ha/replicated-failback-static, however, I am getting this NPE on line 112:
Caused by: java.lang.NullPointerException: Cannot invoke "javax.jms.TextMessage.getText()" because "message0" is null
The NPE seems to indicate that, you cannot call consumer.receive() again even if NOT acknowledged, which seems to contradict the comments on the code. I have run into this same behavior in another example as well, artemis/2.31.2/examples/features/ha/replicated-failback.
Can you please confirm the expected behavior of the example code in the 2.31.2 version? I can then modify the code accordingly. Or please point me to a working sample code.
Thanks,
Lino
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,lIOPl76F2R4EtenIAEJMeQ17kfmHQ6buC7G1W5TRefOcSfxvj_-3dLXV3MKDkCTkac_65Rw0SYmof66i91zGwgb_SS1nKaBpao_Lw7JNNk1mkE33&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Monday, January 8, 2024 9:20 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> Now, the backup process does start processing msgs right away...
What evidence do you have of that?
> ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable.
The log snippets you provided show a reconnection attempt back to dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to dm-activemq-backup-svc:61617 both of which fail almost immediately. Then right after that another connection attempt is made to dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e. from 20:03:44.874 to 20:04:44.933). Are there any other logs during this minute? Did you happen to get any thread dumps during this time? If so, what did they show? If not, could you? What was happening with dm-activemq-live-svc during this time? Perhaps there was something weird going on with the network related to whatever you were doing with that machine to trigger failover.
You might consider simplifying your test-case and then adding complexity bit by bit to see what triggers the difference in behavior. For example, run your test with the client and both brokers on the *same machine*. If that works then move just the client to a remote machine. If that works then test the client just reconnecting to the primary broker upon restart (i.e. no failover). Then finally put all three entities on separate machines and test failover. Hopefully such an exercise will give you a better understanding of where the problem might lie.
Lastly, please indicate what version of ActiveMQ Artemis you're using.
Justin
On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
I’m working with John Lilley on getting an ActiveMQ HA configuration operational. Our current HA config is replication with master and slave instances and with static connections, and it seems to be working, except that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect itself after a failover from the live to the backup instance. Now, the backup process does start processing msgs right away, but the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable. I have included the relevant log traces from the Thread-0 ActiveMQ-client showing the Connection failure detection all the way through when the client is usable again, over 1 minute later. I am hoping that, based on the trace, you can find something that, perhaps, we should be doing differently in the client side?
Here is our Client connection url:
(tcp://dm-activemq-live-svc: 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
Should we also be including the corresponding connector names?
In an earlier email with John Lilley you recommended 2 brokers using “shared storage” instead of “replication”, which I will try next, however, since this issue seems to be on the client-side, I want to make sure our client-side is setup correctly.
Thanks in advance for your help.
Lino
Thread-0 log traces:
2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
---
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616>
2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617<http://10.43.11.76:61617>
2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
---
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616>
2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = ced14fbc
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617<http://10.43.11.76:61617>
2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 10, performing as a failover operation now and recreating objects
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,hqiYs6vnJzfq-X42G9W9ohJtYgn9LRPR-82RHxMyPnKUD5xBwCbo8W6b-PegfmAGwvEENlDO0k2IkoGFFegGxR-H6J72UHHlpEFbGtJzhCF3sucQ&typo=1&ancr_add=1>
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,KX9atZvl9XxpGPWobW9hXsmaLrVekfw2221zvD0PJ9Ik3fEQVyHKcZZCe7MQRciamxD-T-WDEk5Xa7_cAt0PSHwVW3jG7zzhq8XH9XIRC8EXmg,,&typo=1&ancr_add=1>, needed = 65536, arriving = 0
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,ZOCwWoQv34cYLuw9hZvJzBHfSrX6oQeQ_3-SetFMJa1hRC6lXgiKsAElOhfZ_vfnBhRGRJPZjSOqHIbwWO2ltSYYY7ElQjyItp0uDo79_jaDkw,,&typo=1&ancr_add=1>
2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,obbNTaFqn6mglgyZXy1S-0nBLNVtEYlkZBwAEKSGDGN0RuqlR_-7fIQPW4ELNqVFJdenAcKh65C3yQ2b_Lef7W01Tst6ZTrmI66KfHLae572jXv-HF3O&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ytVVTXzfq-J1NXl2ct7T_47F5EbfZNQTXP8dlqw720EiCe32c6CLwlpk0-v4MzUD61FErq0nkRTalFpcTtIN8TDeBvz70rzb__TyeIDm48ZKfFU28qQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,KQYUuEgOaR6V8a_ZC_xBpKZmDneB4LTL1cQzn2Gqtla8ntHJ1fgRs9KelLgvbMd9zvBTg8maXUerYZYQZa-DlfK7fon0JZ0O74PNJ1SOCHqIPN4fZQ6Xv731OGOD&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,OSVh6NSORxz7qZ6eIuUDgWskoYfyXT4NNdjJYGMbH7HeWEUiymys2DnptjYKdmQfqjfHmI-egRZ31yWwd9MKzEd-gbbsuTUsqn_46niKaw,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,DM2boEBGg44zzcd0m37ITkOnfDrkqLE8ZSFGJpEI4AGo2GyOI2yKX08AEkKHxw_UnDMF1vccYEvESFDXjXBBjjIGO5TEsMLWZHbJF3AOIEQ73uc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,wipgGeA7UVZIy1uuqvuxJEx4BA8TvYfSYmO617IgrvnOo-z7sn1vUjXFqIEU9M1HmedBVoHxB_LpvuEZ9p_biioPM8V-V9eS8i8oST46GvBhZZIc&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,0FNTjscT_d2CNIid0J8d_x7C7Peb6_neE-a_MZeKqkM1o_sPGujvpkXuiLv6kms9wSFyYqSux86C1MX2R8WKM8IZBsjKt7sCs03U2lRYlVbw2eTDg0ihSEV1nA,,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3yKPqxpIi9n43QJSCQiysKGlEYttZ5Dmuirc7jUAWNz6orLfIUCYwdDEJ3eU0QiHywsNAJdWqrvnx5rZK6-8TTJb6o8tr1WSvSL7omLF&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,6B4qLrj7q9DzDb0Trp9xgst7cOBhbD4hcEcGyPFul5E_mh3_oGmColB9YQkQTB1gJ6x5Mcm0VwvBLddpDpfFkcZwlzb4YNR9Ym5fcDclFW97pIrQ9oG4&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KqR93A_gEw1FpU7__PsYjSkOC5LH0zFTSKu5K9YlCdVAz3h_c9EkBndO7aliE4RSbK_qslEZp0b-YzJfCOxmB22jAfplsYxguMl1qjC0rWpNjfr9_fh4RjNL3AOK&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,mhChlKkihYjDw_Kk5SL1d_4eb7zhBlSzri_nwjOUsrs6MAvEfDAAjia07yTMUU6CQqc0WbDS2Fn8HJJI0UwJh1N7SKiz9dsP1gJLmgY66huLhXZE9c-3ECGXyIWQ&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,nn8IDmg6XQJLLXzH7kunXDNZa_3A7fUlnHdRCLoha7gQWJO1QeY32duGFAQMYaFHsfIbHbv142mei8HfCqRwrpuUfikYxLkFoXaD3pORZb9roBJs0H8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,DPLEvt-MPmmzv78LEkeXtYHM2q37m12TxX-nqjuIJurQdeXydsSakco9t_O7jgBbQ6y115xxdDb8IonFZbM86zgRDLcvUmeOSJuDmZybD_6JaTX0zpbQ&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,1vNFV-mDaLK92lJo0Mq6jvkCDzuwrSHT4lp1nB6KTzOCSO6comH5xNpEcvtZ_9pc0cQCD2kgDfB5xdxg0tAX2dF5oG5DHY6kxWXOBLnuAwiz1BkEFhw,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,aLydz0SGsclotqCro2upcTg44B_Ij7ckjApLu4B5JUwZKKdMo5z3RyUR4ikHFY8L3t_yB87Zp5atGRwoAPA1L9Mc9c681UjFrG16n-udVjeHtxQNxx8V-Nug&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1mZX-u-XHEZ_e4Ut7YETRvWm7U_qkC-q9QaPeHah4K4yNl5foyM1VSExCtYiisknkelDj6_K2MUHGc65Kd-6-JG_TXLby0OPWodJVjX3BtKaDNBUzusgpzT6ajL7&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-nbEPDxI7J-DFvbJfVVt2Cs1Vi_hN5bw-8vQF3SlM1nthw3VbJLWmS1h73cupUxqBoHCImTzDqI1m5gutx3ZriLpLpU7IqutHCqZuT6CTVWACPtj&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,-lUcdhDzaHWdAWLCc0IxWCFkKtKHxoEd64GWhzXAbeuG_Zm9mvn624BY6uS8MgQn-s2TMF8PbEPYVbOOy6BJuI8ByeiuAF9Y8f2I5wib&typo=1&ancr_add=1>)
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,T2UV-1dji1RODRcU4Ff4j_IzfTH2n47BatzyLZcEOU8VROFr3GS8re7CjDcnnHCSaPEtR-35J3z30dnYcz5K66rVsMVxgDiyh0FvEDKnN67naHjYbn0NpzAc&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
Posted by Justin Bertram <jb...@apache.org>.
Both the "replicated-failback-static" and "replicated-failback" examples
run without issue out of the box on my machine so I'm not sure what's going
on with that. One thing that's odd is that out-of-the-box the line 122 [1]
doesn't match the method call in your NPE. Perhaps you modified the example?
Aside from that there were a handful of questions in my previous message
you didn't address. I think the questions about thread dumps are the most
pressing given that you indicated you're deleting the pod. My guess is that
the client is getting hung in a network call that doesn't want to return
due to the pod disappearing. A series of thread dumps during the minute
delay would confirm or deny that.
Justin
[1]
https://github.com/apache/activemq-artemis/blob/2.31.2/examples/features/ha/replicated-failback-static/src/main/java/org/apache/activemq/artemis/jms/example/ReplicatedFailbackStaticExample.java#L122
On Tue, Jan 9, 2024 at 12:47 PM Lino Pereira
<li...@redpointglobal.com.invalid> wrote:
> Hey Justing, thanks for the help.
>
>
>
> We are using version 2.31.2, which is pretty recent, if not the latest.
>
>
>
> With regards to the log traces, other AMQ threads show similar behavior,
> and our system isn’t doing much until AMQ-client stabilizes.
>
> I am running on a k3d cluster and the failover is triggered by me deleting
> the dm-activemq-live pod, which k8s then spins back up.
>
> Also, our slave config does NOT specify “allow-failback=true”, so the spin
> up of the new dm-activemq-live instance does not trigger a fail-back, which
> is intentional to eliminate failover-failback chaos in such a short time.
>
>
>
> I am trying to setup a simpler test case by using one of the examples,
> artemis/2.31.2/examples/features/ha/replicated-failback-static, however, I
> am getting this NPE on line 112:
>
> Caused by: java.lang.NullPointerException: Cannot invoke
> "javax.jms.TextMessage.getText()" because "message0" is null
>
>
>
> The NPE seems to indicate that, you cannot call consumer.receive() again
> even if NOT acknowledged, which seems to contradict the comments on the
> code. I have run into this same behavior in another example as well,
> artemis/2.31.2/examples/features/ha/replicated-failback.
>
>
>
> Can you please confirm the expected behavior of the example code in the
> 2.31.2 version? I can then modify the code accordingly. Or please point me
> to a working sample code.
>
>
>
> Thanks,
>
> Lino
>
>
>
>
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> Lino Pereira
>
> C++ Developer, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Monday, January 8, 2024 9:20 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Looking for HA/replication boilerplate broker.xml and
> advice (continued)
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> > Now, the backup process does start processing msgs right away...
>
>
>
> What evidence do you have of that?
>
>
>
> > ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute
> apart, before it stabilizes after the failover and is once again usable.
>
>
>
> The log snippets you provided show a reconnection attempt back to
> dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to
> dm-activemq-backup-svc:61617 both of which fail almost immediately. Then
> right after that another connection attempt is made to
> dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e.
> from 20:03:44.874 to 20:04:44.933). Are there any other logs during this
> minute? Did you happen to get any thread dumps during this time? If so,
> what did they show? If not, could you? What was happening with
> dm-activemq-live-svc during this time? Perhaps there was something weird
> going on with the network related to whatever you were doing with that
> machine to trigger failover.
>
>
>
> You might consider simplifying your test-case and then adding complexity
> bit by bit to see what triggers the difference in behavior. For example,
> run your test with the client and both brokers on the *same machine*. If
> that works then move just the client to a remote machine. If that works
> then test the client just reconnecting to the primary broker upon restart
> (i.e. no failover). Then finally put all three entities on separate
> machines and test failover. Hopefully such an exercise will give you a
> better understanding of where the problem might lie.
>
>
>
> Lastly, please indicate what version of ActiveMQ Artemis you're using.
>
>
>
>
>
> Justin
>
>
>
> On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira <
> lino.pereira@redpointglobal.com.invalid> wrote:
>
> Hi Justin,
>
> I’m working with John Lilley on getting an ActiveMQ HA configuration
> operational. Our current HA config is replication with master and slave
> instances and with static connections, and it seems to be working, except
> that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect
> itself after a failover from the live to the backup instance. Now, the
> backup process does start processing msgs right away, but the
> ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart,
> before it stabilizes after the failover and is once again usable. I have
> included the relevant log traces from the Thread-0 ActiveMQ-client showing
> the Connection failure detection all the way through when the client is
> usable again, over 1 minute later. I am hoping that, based on the trace,
> you can find something that, perhaps, we should be doing differently in the
> client side?
>
>
>
> Here is our Client connection url:
>
> (tcp://dm-activemq-live-svc:
> 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
>
>
>
> Should we also be including the corresponding connector names?
>
>
>
> In an earlier email with John Lilley you recommended 2 brokers using
> “shared storage” instead of “replication”, which I will try next, however,
> since this issue seems to be on the client-side, I want to make sure our
> client-side is setup correctly.
>
>
>
> Thanks in advance for your help.
>
> Lino
>
>
>
> Thread-0 log traces:
>
> 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-live-svc/10.43.197.248:61616 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=a8a4a77c, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> ---
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and currentConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.197.248:61616
>
>
>
> 2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.11.76:61617
>
>
>
> 2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> ---
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and currentConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.197.248:61616
>
> 2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = ced14fbc
>
>
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.11.76:61617
>
>
>
> 2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
>
>
> 2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 10, performing as a failover operation now and recreating
> objects
>
>
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address
> activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,hqiYs6vnJzfq-X42G9W9ohJtYgn9LRPR-82RHxMyPnKUD5xBwCbo8W6b-PegfmAGwvEENlDO0k2IkoGFFegGxR-H6J72UHHlpEFbGtJzhCF3sucQ&typo=1&ancr_add=1>
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,KX9atZvl9XxpGPWobW9hXsmaLrVekfw2221zvD0PJ9Ik3fEQVyHKcZZCe7MQRciamxD-T-WDEk5Xa7_cAt0PSHwVW3jG7zzhq8XH9XIRC8EXmg,,&typo=1&ancr_add=1>,
> needed = 65536, arriving = 0
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536
> credits on address activemq.management
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,ZOCwWoQv34cYLuw9hZvJzBHfSrX6oQeQ_3-SetFMJa1hRC6lXgiKsAElOhfZ_vfnBhRGRJPZjSOqHIbwWO2ltSYYY7ElQjyItp0uDo79_jaDkw,,&typo=1&ancr_add=1>
>
> 2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=a8a4a77c, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(
> ClientSessionFactoryImpl.java:410
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,obbNTaFqn6mglgyZXy1S-0nBLNVtEYlkZBwAEKSGDGN0RuqlR_-7fIQPW4ELNqVFJdenAcKh65C3yQ2b_Lef7W01Tst6ZTrmI66KfHLae572jXv-HF3O&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(
> NettyConnector.java:835
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ytVVTXzfq-J1NXl2ct7T_47F5EbfZNQTXP8dlqw720EiCe32c6CLwlpk0-v4MzUD61FErq0nkRTalFpcTtIN8TDeBvz70rzb__TyeIDm48ZKfFU28qQ,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(
> ClientSessionFactoryImpl.java:666
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,KQYUuEgOaR6V8a_ZC_xBpKZmDneB4LTL1cQzn2Gqtla8ntHJ1fgRs9KelLgvbMd9zvBTg8maXUerYZYQZa-DlfK7fon0JZ0O74PNJ1SOCHqIPN4fZQ6Xv731OGOD&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(
> ClientSessionFactoryImpl.java:566
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,OSVh6NSORxz7qZ6eIuUDgWskoYfyXT4NNdjJYGMbH7HeWEUiymys2DnptjYKdmQfqjfHmI-egRZ31yWwd9MKzEd-gbbsuTUsqn_46niKaw,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(
> ClientSessionFactoryImpl.java:1407
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,DM2boEBGg44zzcd0m37ITkOnfDrkqLE8ZSFGJpEI4AGo2GyOI2yKX08AEkKHxw_UnDMF1vccYEvESFDXjXBBjjIGO5TEsMLWZHbJF3AOIEQ73uc,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(
> AbstractRemotingConnection.java:98
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,wipgGeA7UVZIy1uuqvuxJEx4BA8TvYfSYmO617IgrvnOo-z7sn1vUjXFqIEU9M1HmedBVoHxB_LpvuEZ9p_biioPM8V-V9eS8i8oST46GvBhZZIc&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,0FNTjscT_d2CNIid0J8d_x7C7Peb6_neE-a_MZeKqkM1o_sPGujvpkXuiLv6kms9wSFyYqSux86C1MX2R8WKM8IZBsjKt7sCs03U2lRYlVbw2eTDg0ihSEV1nA,,&typo=1&ancr_add=1>
> (RemotingConnectionImpl.java:212
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3yKPqxpIi9n43QJSCQiysKGlEYttZ5Dmuirc7jUAWNz6orLfIUCYwdDEJ3eU0QiHywsNAJdWqrvnx5rZK6-8TTJb6o8tr1WSvSL7omLF&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$
> CloseRunnable.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,6B4qLrj7q9DzDb0Trp9xgst7cOBhbD4hcEcGyPFul5E_mh3_oGmColB9YQkQTB1gJ6x5Mcm0VwvBLddpDpfFkcZwlzb4YNR9Ym5fcDclFW97pIrQ9oG4&typo=1&ancr_add=1>
> (ClientSessionFactoryImpl.java:1172
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KqR93A_gEw1FpU7__PsYjSkOC5LH0zFTSKu5K9YlCdVAz3h_c9EkBndO7aliE4RSbK_qslEZp0b-YzJfCOxmB22jAfplsYxguMl1qjC0rWpNjfr9_fh4RjNL3AOK&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:57
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,mhChlKkihYjDw_Kk5SL1d_4eb7zhBlSzri_nwjOUsrs6MAvEfDAAjia07yTMUU6CQqc0WbDS2Fn8HJJI0UwJh1N7SKiz9dsP1gJLmgY66huLhXZE9c-3ECGXyIWQ&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(
> OrderedExecutor.java:32
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,nn8IDmg6XQJLLXzH7kunXDNZa_3A7fUlnHdRCLoha7gQWJO1QeY32duGFAQMYaFHsfIbHbv142mei8HfCqRwrpuUfikYxLkFoXaD3pORZb9roBJs0H8,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(
> ProcessorBase.java:68
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,DPLEvt-MPmmzv78LEkeXtYHM2q37m12TxX-nqjuIJurQdeXydsSakco9t_O7jgBbQ6y115xxdDb8IonFZbM86zgRDLcvUmeOSJuDmZybD_6JaTX0zpbQ&typo=1&ancr_add=1>
> )
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1136
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,1vNFV-mDaLK92lJo0Mq6jvkCDzuwrSHT4lp1nB6KTzOCSO6comH5xNpEcvtZ_9pc0cQCD2kgDfB5xdxg0tAX2dF5oG5DHY6kxWXOBLnuAwiz1BkEFhw,&typo=1&ancr_add=1>
> )
>
> at java.base/java.util.concurrent.ThreadPoolExecutor$
> Worker.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,aLydz0SGsclotqCro2upcTg44B_Ij7ckjApLu4B5JUwZKKdMo5z3RyUR4ikHFY8L3t_yB87Zp5atGRwoAPA1L9Mc9c681UjFrG16n-udVjeHtxQNxx8V-Nug&typo=1&ancr_add=1>
> (ThreadPoolExecutor.java:635
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1mZX-u-XHEZ_e4Ut7YETRvWm7U_qkC-q9QaPeHah4K4yNl5foyM1VSExCtYiisknkelDj6_K2MUHGc65Kd-6-JG_TXLby0OPWodJVjX3BtKaDNBUzusgpzT6ajL7&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$
> 1.run
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-nbEPDxI7J-DFvbJfVVt2Cs1Vi_hN5bw-8vQF3SlM1nthw3VbJLWmS1h73cupUxqBoHCImTzDqI1m5gutx3ZriLpLpU7IqutHCqZuT6CTVWACPtj&typo=1&ancr_add=1>
> (ActiveMQThreadFactory.java:118
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,-lUcdhDzaHWdAWLCc0IxWCFkKtKHxoEd64GWhzXAbeuG_Zm9mvn624BY6uS8MgQn-s2TMF8PbEPYVbOOy6BJuI8ByeiuAF9Y8f2I5wib&typo=1&ancr_add=1>
> )
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,T2UV-1dji1RODRcU4Ff4j_IzfTH2n47BatzyLZcEOU8VROFr3GS8re7CjDcnnHCSaPEtR-35J3z30dnYcz5K66rVsMVxgDiyh0FvEDKnN67naHjYbn0NpzAc&typo=1>
>
> *Lino Pereira *
>
> *C++ Developer, Redpoint Global Inc. *
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
RE: Looking for HA/replication boilerplate broker.xml and advice (continued)
Posted by Lino Pereira <li...@redpointglobal.com.INVALID>.
Hey Justing, thanks for the help.
We are using version 2.31.2, which is pretty recent, if not the latest.
With regards to the log traces, other AMQ threads show similar behavior, and our system isn’t doing much until AMQ-client stabilizes.
I am running on a k3d cluster and the failover is triggered by me deleting the dm-activemq-live pod, which k8s then spins back up.
Also, our slave config does NOT specify “allow-failback=true”, so the spin up of the new dm-activemq-live instance does not trigger a fail-back, which is intentional to eliminate failover-failback chaos in such a short time.
I am trying to setup a simpler test case by using one of the examples, artemis/2.31.2/examples/features/ha/replicated-failback-static, however, I am getting this NPE on line 112:
Caused by: java.lang.NullPointerException: Cannot invoke "javax.jms.TextMessage.getText()" because "message0" is null
The NPE seems to indicate that, you cannot call consumer.receive() again even if NOT acknowledged, which seems to contradict the comments on the code. I have run into this same behavior in another example as well, artemis/2.31.2/examples/features/ha/replicated-failback.
Can you please confirm the expected behavior of the example code in the 2.31.2 version? I can then modify the code accordingly. Or please point me to a working sample code.
Thanks,
Lino
[rg] <https://www.redpointglobal.com/>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>
Sent: Monday, January 8, 2024 9:20 PM
To: users@activemq.apache.org
Subject: Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
> Now, the backup process does start processing msgs right away...
What evidence do you have of that?
> ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable.
The log snippets you provided show a reconnection attempt back to dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to dm-activemq-backup-svc:61617 both of which fail almost immediately. Then right after that another connection attempt is made to dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e. from 20:03:44.874 to 20:04:44.933). Are there any other logs during this minute? Did you happen to get any thread dumps during this time? If so, what did they show? If not, could you? What was happening with dm-activemq-live-svc during this time? Perhaps there was something weird going on with the network related to whatever you were doing with that machine to trigger failover.
You might consider simplifying your test-case and then adding complexity bit by bit to see what triggers the difference in behavior. For example, run your test with the client and both brokers on the *same machine*. If that works then move just the client to a remote machine. If that works then test the client just reconnecting to the primary broker upon restart (i.e. no failover). Then finally put all three entities on separate machines and test failover. Hopefully such an exercise will give you a better understanding of where the problem might lie.
Lastly, please indicate what version of ActiveMQ Artemis you're using.
Justin
On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira <li...@redpointglobal.com.invalid>> wrote:
Hi Justin,
I’m working with John Lilley on getting an ActiveMQ HA configuration operational. Our current HA config is replication with master and slave instances and with static connections, and it seems to be working, except that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect itself after a failover from the live to the backup instance. Now, the backup process does start processing msgs right away, but the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart, before it stabilizes after the failover and is once again usable. I have included the relevant log traces from the Thread-0 ActiveMQ-client showing the Connection failure detection all the way through when the client is usable again, over 1 minute later. I am hoping that, based on the trace, you can find something that, perhaps, we should be doing differently in the client side?
Here is our Client connection url:
(tcp://dm-activemq-live-svc: 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
Should we also be including the corresponding connector names?
In an earlier email with John Lilley you recommended 2 brokers using “shared storage” instead of “replication”, which I will try next, however, since this issue seems to be on the client-side, I want to make sure our client-side is setup correctly.
Thanks in advance for your help.
Lino
Thread-0 log traces:
2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection failure to dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616> has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
---
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 0/200
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616>
2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617<http://10.43.11.76:61617>
2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no connection been made, returning null
---
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying reconnection attempt 1/200
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and currentConnectorConfig: TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-live-svc:61616
2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-live-svc/10.43.197.248:61616<http://10.43.197.248:61616>
2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)] NettyConnector.initChannel:762 [] DEBUG - Added ActiveMQClientChannelHandler to Channel with id = ced14fbc
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector towards NettyConnector [host=dm-activemq-live-svc, port=61616, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying to connect with connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0<ma...@68dd9fb0> and backupConnectorConfig: TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:553 [] DEBUG - Connector NettyConnector [host=dm-activemq-backup-svc, port=61617, httpEnabled=false, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using nio
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version 4.1.101.Final to dm-activemq-backup-svc:61617
2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)] NettyConnector.createConnection:877 [] DEBUG - Remote destination: dm-activemq-backup-svc/10.43.11.76:61617<http://10.43.11.76:61617>
2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG - Connected with the backupConnectorConfig=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61617&host=dm-activemq-backup-svc
2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection successful
2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 11, performing as a failover operation now and recreating objects
2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)] ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't reattach session 10, performing as a failover operation now and recreating objects
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,hqiYs6vnJzfq-X42G9W9ohJtYgn9LRPR-82RHxMyPnKUD5xBwCbo8W6b-PegfmAGwvEENlDO0k2IkoGFFegGxR-H6J72UHHlpEFbGtJzhCF3sucQ&typo=1&ancr_add=1>
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,KX9atZvl9XxpGPWobW9hXsmaLrVekfw2221zvD0PJ9Ik3fEQVyHKcZZCe7MQRciamxD-T-WDEk5Xa7_cAt0PSHwVW3jG7zzhq8XH9XIRC8EXmg,,&typo=1&ancr_add=1>, needed = 65536, arriving = 0
2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)] AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536 credits on address activemq.management<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2factivemq.management&c=E,1,ZOCwWoQv34cYLuw9hZvJzBHfSrX6oQeQ_3-SetFMJa1hRC6lXgiKsAElOhfZ_vfnBhRGRJPZjSOqHIbwWO2ltSYYY7ElQjyItp0uDo79_jaDkw,,&typo=1&ancr_add=1>
2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)] ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure captured on connectionID=a8a4a77c, performing failover or reconnection now ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: Channel disconnected]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a410&c=E,1,obbNTaFqn6mglgyZXy1S-0nBLNVtEYlkZBwAEKSGDGN0RuqlR_-7fIQPW4ELNqVFJdenAcKh65C3yQ2b_Lef7W01Tst6ZTrmI66KfHLae572jXv-HF3O&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fNettyConnector.java%3a835&c=E,1,ytVVTXzfq-J1NXl2ct7T_47F5EbfZNQTXP8dlqw720EiCe32c6CLwlpk0-v4MzUD61FErq0nkRTalFpcTtIN8TDeBvz70rzb__TyeIDm48ZKfFU28qQ,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a666&c=E,1,KQYUuEgOaR6V8a_ZC_xBpKZmDneB4LTL1cQzn2Gqtla8ntHJ1fgRs9KelLgvbMd9zvBTg8maXUerYZYQZa-DlfK7fon0JZ0O74PNJ1SOCHqIPN4fZQ6Xv731OGOD&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a566&c=E,1,OSVh6NSORxz7qZ6eIuUDgWskoYfyXT4NNdjJYGMbH7HeWEUiymys2DnptjYKdmQfqjfHmI-egRZ31yWwd9MKzEd-gbbsuTUsqn_46niKaw,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1407&c=E,1,DM2boEBGg44zzcd0m37ITkOnfDrkqLE8ZSFGJpEI4AGo2GyOI2yKX08AEkKHxw_UnDMF1vccYEvESFDXjXBBjjIGO5TEsMLWZHbJF3AOIEQ73uc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fAbstractRemotingConnection.java%3a98&c=E,1,wipgGeA7UVZIy1uuqvuxJEx4BA8TvYfSYmO617IgrvnOo-z7sn1vUjXFqIEU9M1HmedBVoHxB_LpvuEZ9p_biioPM8V-V9eS8i8oST46GvBhZZIc&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2forg.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail&c=E,1,0FNTjscT_d2CNIid0J8d_x7C7Peb6_neE-a_MZeKqkM1o_sPGujvpkXuiLv6kms9wSFyYqSux86C1MX2R8WKM8IZBsjKt7sCs03U2lRYlVbw2eTDg0ihSEV1nA,,&typo=1&ancr_add=1>(RemotingConnectionImpl.java:212<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRemotingConnectionImpl.java%3a212&c=E,1,3yKPqxpIi9n43QJSCQiysKGlEYttZ5Dmuirc7jUAWNz6orLfIUCYwdDEJ3eU0QiHywsNAJdWqrvnx5rZK6-8TTJb6o8tr1WSvSL7omLF&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCloseRunnable.run&c=E,1,6B4qLrj7q9DzDb0Trp9xgst7cOBhbD4hcEcGyPFul5E_mh3_oGmColB9YQkQTB1gJ6x5Mcm0VwvBLddpDpfFkcZwlzb4YNR9Ym5fcDclFW97pIrQ9oG4&typo=1&ancr_add=1>(ClientSessionFactoryImpl.java:1172<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1172&c=E,1,KqR93A_gEw1FpU7__PsYjSkOC5LH0zFTSKu5K9YlCdVAz3h_c9EkBndO7aliE4RSbK_qslEZp0b-YzJfCOxmB22jAfplsYxguMl1qjC0rWpNjfr9_fh4RjNL3AOK&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a57&c=E,1,mhChlKkihYjDw_Kk5SL1d_4eb7zhBlSzri_nwjOUsrs6MAvEfDAAjia07yTMUU6CQqc0WbDS2Fn8HJJI0UwJh1N7SKiz9dsP1gJLmgY66huLhXZE9c-3ECGXyIWQ&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a32&c=E,1,nn8IDmg6XQJLLXzH7kunXDNZa_3A7fUlnHdRCLoha7gQWJO1QeY32duGFAQMYaFHsfIbHbv142mei8HfCqRwrpuUfikYxLkFoXaD3pORZb9roBJs0H8,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a68&c=E,1,DPLEvt-MPmmzv78LEkeXtYHM2q37m12TxX-nqjuIJurQdeXydsSakco9t_O7jgBbQ6y115xxdDb8IonFZbM86zgRDLcvUmeOSJuDmZybD_6JaTX0zpbQ&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1136&c=E,1,1vNFV-mDaLK92lJo0Mq6jvkCDzuwrSHT4lp1nB6KTzOCSO6comH5xNpEcvtZ_9pc0cQCD2kgDfB5xdxg0tAX2dF5oG5DHY6kxWXOBLnuAwiz1BkEFhw,&typo=1&ancr_add=1>)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,aLydz0SGsclotqCro2upcTg44B_Ij7ckjApLu4B5JUwZKKdMo5z3RyUR4ikHFY8L3t_yB87Zp5atGRwoAPA1L9Mc9c681UjFrG16n-udVjeHtxQNxx8V-Nug&typo=1&ancr_add=1>(ThreadPoolExecutor.java:635<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a635&c=E,1,1mZX-u-XHEZ_e4Ut7YETRvWm7U_qkC-q9QaPeHah4K4yNl5foyM1VSExCtYiisknkelDj6_K2MUHGc65Kd-6-JG_TXLby0OPWodJVjX3BtKaDNBUzusgpzT6ajL7&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,-nbEPDxI7J-DFvbJfVVt2Cs1Vi_hN5bw-8vQF3SlM1nthw3VbJLWmS1h73cupUxqBoHCImTzDqI1m5gutx3ZriLpLpU7IqutHCqZuT6CTVWACPtj&typo=1&ancr_add=1>(ActiveMQThreadFactory.java:118<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,-lUcdhDzaHWdAWLCc0IxWCFkKtKHxoEd64GWhzXAbeuG_Zm9mvn624BY6uS8MgQn-s2TMF8PbEPYVbOOy6BJuI8ByeiuAF9Y8f2I5wib&typo=1&ancr_add=1>)
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,T2UV-1dji1RODRcU4Ff4j_IzfTH2n47BatzyLZcEOU8VROFr3GS8re7CjDcnnHCSaPEtR-35J3z30dnYcz5K66rVsMVxgDiyh0FvEDKnN67naHjYbn0NpzAc&typo=1>
Lino Pereira
C++ Developer, Redpoint Global Inc.
34 Washington Street, Suite 205 Wellesley Hills, MA 02481
lino.pereira@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Looking for HA/replication boilerplate broker.xml and advice (continued)
Posted by Justin Bertram <jb...@apache.org>.
> Now, the backup process does start processing msgs right away...
What evidence do you have of that?
> ...the ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute
apart, before it stabilizes after the failover and is once again usable.
The log snippets you provided show a reconnection attempt back to
dm-activemq-live-svc:61616 (i.e. the broker which failed) and then to
dm-activemq-backup-svc:61617 both of which fail almost immediately. Then
right after that another connection attempt is made to
dm-activemq-live-svc:61616 which appears to take a minute to fail (i.e.
from 20:03:44.874 to 20:04:44.933). Are there any other logs during this
minute? Did you happen to get any thread dumps during this time? If so,
what did they show? If not, could you? What was happening with
dm-activemq-live-svc during this time? Perhaps there was something weird
going on with the network related to whatever you were doing with that
machine to trigger failover.
You might consider simplifying your test-case and then adding complexity
bit by bit to see what triggers the difference in behavior. For example,
run your test with the client and both brokers on the *same machine*. If
that works then move just the client to a remote machine. If that works
then test the client just reconnecting to the primary broker upon restart
(i.e. no failover). Then finally put all three entities on separate
machines and test failover. Hopefully such an exercise will give you a
better understanding of where the problem might lie.
Lastly, please indicate what version of ActiveMQ Artemis you're using.
Justin
On Mon, Jan 8, 2024 at 7:43 PM Lino Pereira
<li...@redpointglobal.com.invalid> wrote:
> Hi Justin,
>
> I’m working with John Lilley on getting an ActiveMQ HA configuration
> operational. Our current HA config is replication with master and slave
> instances and with static connections, and it seems to be working, except
> that the ActiveMQ-client is taking over 1 minute to reconfigure/reconnect
> itself after a failover from the live to the backup instance. Now, the
> backup process does start processing msgs right away, but the
> ActiveMQ-client needs 2 “Trying reconnection attempts”, 1-minute apart,
> before it stabilizes after the failover and is once again usable. I have
> included the relevant log traces from the Thread-0 ActiveMQ-client showing
> the Connection failure detection all the way through when the client is
> usable again, over 1 minute later. I am hoping that, based on the trace,
> you can find something that, perhaps, we should be doing differently in the
> client side?
>
>
>
> Here is our Client connection url:
>
> (tcp://dm-activemq-live-svc:
> 61616,tcp://dm-activemq-backup-svc:61617)?ha=true&reconnectAttempts=100&initialConnectAttempts=100
>
>
>
> Should we also be including the corresponding connector names?
>
>
>
> In an earlier email with John Lilley you recommended 2 brokers using
> “shared storage” instead of “replication”, which I will try next, however,
> since this issue seems to be on the client-side, I want to make sure our
> client-side is setup correctly.
>
>
>
> Thanks in advance for your help.
>
> Lino
>
>
>
> Thread-0 log traces:
>
> 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection
> failure to dm-activemq-live-svc/10.43.197.248:61616 has been detected:
> AMQ219015: The connection was disconnected because of server shutdown
> [code=DISCONNECTED]
>
> 2024-01-08T20:03:43.852 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=a8a4a77c, performing failover or reconnection now
> ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The
> connection was disconnected because of server shutdown]
>
> ---
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 0/200
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and currentConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-08T20:03:43.857 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.197.248:61616
>
>
>
> 2024-01-08T20:03:43.864 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-08T20:03:43.866 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.11.76:61617
>
>
>
> 2024-01-08T20:03:43.872 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-backup-svc, port=61617,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:03:43.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1297 [] DEBUG - no
> connection been made, returning null
>
> ---
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:956 [] DEBUG - Trying
> reconnection attempt 1/200
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and currentConnectorConfig: TransportConfiguration(name=null,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61616&host=dm-activemq-live-svc&reconnectAttempts=200&ha=true&initialConnectAttempts=200
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-live-svc, port=61616, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-live-svc:61616
>
> 2024-01-08T20:03:44.873 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-live-svc/10.43.197.248:61616
>
> 2024-01-08T20:03:44.874 [Thread-8 (ActiveMQ-client-netty-threads)]
> NettyConnector.initChannel:762 [] DEBUG - Added
> ActiveMQClientChannelHandler to Channel with id = ced14fbc
>
>
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.openTransportConnection:1215 [] DEBUG - Connector
> towards NettyConnector [host=dm-activemq-live-svc, port=61616,
> httpEnabled=false, httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> failed
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1328 [] DEBUG - Trying
> to connect with
> connectorFactory=org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@68dd9fb0
> and backupConnectorConfig: TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:553 [] DEBUG - Connector NettyConnector
> [host=dm-activemq-backup-svc, port=61617, httpEnabled=false,
> httpUpgradeEnabled=false, useServlet=false,
> servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true]
> using nio
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.start:772 [] DEBUG - Started NIO Netty Connector version
> 4.1.101.Final to dm-activemq-backup-svc:61617
>
> 2024-01-08T20:04:44.933 [Thread-0 (ActiveMQ-client-global-threads)]
> NettyConnector.createConnection:877 [] DEBUG - Remote destination:
> dm-activemq-backup-svc/10.43.11.76:61617
>
>
>
> 2024-01-08T20:04:44.936 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.createTransportConnection:1336 [] DEBUG -
> Connected with the
> backupConnectorConfig=TransportConfiguration(name=netty-connector,
> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
> ?port=61617&host=dm-activemq-backup-svc
>
>
>
> 2024-01-08T20:04:44.966 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.getConnectionWithRetry:964 [] DEBUG - Reconnection
> successful
>
> 2024-01-08T20:04:44.969 [Thread-0 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 11, performing as a failover operation now and recreating
> objects
>
> 2024-01-08T20:04:45.002 [Thread-0 (ActiveMQ-client-global-threads)]
> ActiveMQSessionContext.reattachOnNewConnection:888 [] DEBUG - Couldn't
> reattach session 10, performing as a failover operation now and recreating
> objects
>
>
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.reset:123 [] DEBUG - reset credits on address
> activemq.management
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.checkCredits:178 [] DEBUG - Requesting 65536
> credits on address activemq.management, needed = 65536, arriving = 0
>
> 2024-01-08T20:04:45.019 [Thread-0 (ActiveMQ-client-global-threads)]
> AbstractProducerCreditsImpl.requestCredits:187 [] DEBUG - Request 65536
> credits on address activemq.management
>
> 2024-01-08T20:04:45.020 [Thread-0 (ActiveMQ-client-global-threads)]
> ClientSessionFactoryImpl.failoverOrReconnect:587 [] DEBUG - Failure
> captured on connectionID=a8a4a77c, performing failover or reconnection now
> ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
> Channel disconnected]
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:410)
>
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.close(NettyConnector.java:835)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:666)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:566)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1407)
>
> at
> org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98)
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:212)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1172)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
>
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
>
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
> [image: rg] <https://www.redpointglobal.com/>
>
> Lino Pereira
>
> C++ Developer, Redpoint Global Inc.
>
> 34 Washington Street, Suite 205 Wellesley Hills, MA 02481
>
> lino.pereira@redpointglobal.com
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>