You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Timothy A. Bish (Jira)" <ji...@apache.org> on 2025/06/06 14:41:00 UTC
[jira] [Reopened] (ARTEMIS-5377) Deadlock and Crash of Artemis Broker When Restarting a Bridged Peer
[ https://issues.apache.org/jira/browse/ARTEMIS-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy A. Bish reopened ARTEMIS-5377:
--------------------------------------
> Deadlock and Crash of Artemis Broker When Restarting a Bridged Peer
> -------------------------------------------------------------------
>
> Key: ARTEMIS-5377
> URL: https://issues.apache.org/jira/browse/ARTEMIS-5377
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker, Clustering
> Affects Versions: 2.31.2
> Reporter: Alexander
> Priority: Major
> Attachments: falled_broker.log, restarder_boker.log
>
>
> I encountered a similar issue as described on ARTEMIS-4114.
> Configuration:
> {code:xml}
> <ha-policy>
> <live-only>
> <scale-down>
> <connectors>
> <connector-ref>ART.GW.CLS1-connector</connector-ref>
> <connector-ref>ART.GW.CLS2-connector</connector-ref>
> <connector-ref>ART.GW.CLS3-connector</connector-ref>
> </connectors>
> </scale-down>
> </live-only>
> </ha-policy>
> ...
> <bridges>
> <bridge name="ART.EL.CLS">
> <queue-name>ART.EL.CLS</queue-name>
> <forwarding-address>ART.EL.CLS</forwarding-address>
> <reconnect-attempts>-1</reconnect-attempts>
> <static-connectors>
> <connector-ref>ART.EL.CLS1-connector</connector-ref>
> <connector-ref>ART.EL.CLS2-connector</connector-ref>
> <connector-ref>ART.EL.CLS3-connector</connector-ref>
> <connector-ref>ART.EL.CLS4-connector</connector-ref>
> <connector-ref>ART.EL.CLS5connector</connector-ref>
> <connector-ref>ART.EL.CLS6-connector</connector-ref>
> </static-connectors>
> </bridge>
> ... {code}
>
> When restarting one broker, another broker connected to it via a bridge crashes with a deadlock error. Before the crash, the following warnings appeared on the connected broker:
> {noformat}
> 2025-03-08 02:39:09,089 WARN [org.apache.activemq.artemis.core.server] AMQ222094: Bridge unable to send message Reference[314223917984]:NON-RELIABLE:CoreMessage[...]@2117085983, will try again once bridge reconnects ActiveMQObjectClosedException[errorType=OBJECT_CLOSED message=AMQ219018: Producer is closed]
> at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.checkClosed(ClientProducerImpl.java:310)
> at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:127)
> at org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl.deliverStandardMessage(BridgeImpl.java:767)
> at org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:628)
> at org.apache.activemq.artemis.core.server.impl.QueueImpl.handle(QueueImpl.java:4055)
> at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:3191)
> at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4380)
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
> 2025-03-08 02:39:09,089 WARN [org.apache.activemq.artemis.core.server] AMQ222095: Connection failed with failedOver=false
> 2025-03-08 02:39:09,089 WARN [org.apache.activemq.artemis.core.server.impl.QueueImpl] null java.util.NoSuchElementException
> at org.apache.activemq.artemis.utils.collections.PriorityLinkedListImpl$PriorityLinkedListIterator.repeat(PriorityLinkedListImpl.java:225)
> at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:3214)
> at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4380)
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118){noformat}
> I have attached the full log. From it, you can see that at 02:41:21 Broker 1 completely shut down. Then, at 02:41:38 Broker 2 also stopped with the following error:
> {noformat}
> 2025-03-08 02:41:38,069 INFO [org.apache.activemq.artemis.core.server] AMQ224107: The Critical Analyzer detected slow paths on the broker. It is recommended that you enable trace logs on org.apache.activemq.artemis.utils.critical while you troubleshoot this issue. You should disable the trace logs when you have finished troubleshooting.
> 2025-03-08 02:41:38,069 ERROR [org.apache.activemq.artemis.core.server] AMQ224079: The process for the virtual machine will be killed, as component QueueImpl[name=ART.EL.CLS, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=ART.GW.CLS7], temp=false]@3789b8fe is not responsive
> 2025-03-08 02:41:40,152 WARN [org.apache.activemq.artemis.core.server] AMQ222199: Thread dump: *******************************************************************************
> Complete Thread dump
> "Reference Handler" Id=2 RUNNABLE
> at java.base@11.0.2/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
> at java.base@11.0.2/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
> at java.base@11.0.2/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
>
>
> "Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@279208e0
> at java.base@11.0.2/java.lang.Object.wait(Native Method)
> - waiting on java.lang.ref.ReferenceQueue$Lock@279208e0
> at java.base@11.0.2/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
> at java.base@11.0.2/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
> at java.base@11.0.2/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
>
>
> "Signal Dispatcher" Id=4 RUNNABLE
> ...{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@activemq.apache.org
For additional commands, e-mail: issues-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact