You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Justin Bertram (Jira)" <ji...@apache.org> on 2024/09/20 20:13:00 UTC
[jira] [Comment Edited] (ARTEMIS-5042) The load balancing is not working correctly when several brokers are down
[ https://issues.apache.org/jira/browse/ARTEMIS-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883311#comment-17883311 ]
Justin Bertram edited comment on ARTEMIS-5042 at 9/20/24 8:12 PM:
------------------------------------------------------------------
I believe the problem is being caused by this exception which is being thrown and short-circuiting the removal of the cluster bridge flow record (resulting in messages still being routed to the bridge's store-and-forward queue):
{noformat}
2024-09-20 10:34:50,998 WARN [org.apache.activemq.artemis.core.server] AMQ222268: Failed to remove a record
java.lang.IllegalStateException: Cannot find binding mtmccon_ucpmula_sms_193#mcla_simusmsc3_ucpmula_queue9eb3fb16-71d6-11ef-9298-506b8daae603
at org.apache.activemq.artemis.core.postoffice.impl.SimpleAddressManager.removeBindingInternal(SimpleAddressManager.java:228) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.postoffice.impl.WildcardAddressManager.lambda$removeBinding$2(WildcardAddressManager.java:129) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.postoffice.impl.AddressPartNode.visitValues(AddressPartNode.java:271) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.postoffice.impl.AddressPartNode.visitNonWildcard(AddressPartNode.java:219) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.postoffice.impl.AddressMap.visitMatching(AddressMap.java:64) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.postoffice.impl.WildcardAddressManager.removeBinding(WildcardAddressManager.java:129) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.removeBinding(PostOfficeImpl.java:988) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.removeBinding(ClusterConnectionImpl.java:1384) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.clearBindings(ClusterConnectionImpl.java:1249) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.close(ClusterConnectionImpl.java:1052) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl.removeRecord(ClusterConnectionImpl.java:1619) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionBridge.fail(ClusterConnectionBridge.java:436) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl.connectionFailed(BridgeImpl.java:699) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:878) ~[artemis-core-client-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:804) ~[artemis-core-client-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:576) ~[artemis-core-client-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1417) ~[artemis-core-client-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98) ~[artemis-core-client-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:209) ~[artemis-core-client-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1182) ~[artemis-core-client-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57) ~[artemis-commons-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) ~[artemis-commons-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) ~[artemis-commons-2.35.0.jar:2.35.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.35.0.jar:2.35.0]{noformat}
I think this logic could be improved to simply log when a binding can't be found in this context. I'll send a PR for this.
That said, this is not normal behavior, and I haven't been able to reproduce this in my local tests.
was (Author: jbertram):
The problem is being caused by this exception which is being thrown and short-circuiting the removal of the cluster bridge flow record (resulting in messages still being routed to the bridge's store-and-forward queue):
{noformat}
2024-09-20 10:34:50,998 WARN [org.apache.activemq.artemis.core.server] AMQ222268: Failed to remove a record
java.lang.IllegalStateException: Cannot find binding mtmccon_ucpmula_sms_193#mcla_simusmsc3_ucpmula_queue9eb3fb16-71d6-11ef-9298-506b8daae603
at org.apache.activemq.artemis.core.postoffice.impl.SimpleAddressManager.removeBindingInternal(SimpleAddressManager.java:228) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.postoffice.impl.WildcardAddressManager.lambda$removeBinding$2(WildcardAddressManager.java:129) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.postoffice.impl.AddressPartNode.visitValues(AddressPartNode.java:271) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.postoffice.impl.AddressPartNode.visitNonWildcard(AddressPartNode.java:219) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.postoffice.impl.AddressMap.visitMatching(AddressMap.java:64) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.postoffice.impl.WildcardAddressManager.removeBinding(WildcardAddressManager.java:129) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.removeBinding(PostOfficeImpl.java:988) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.removeBinding(ClusterConnectionImpl.java:1384) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.clearBindings(ClusterConnectionImpl.java:1249) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.close(ClusterConnectionImpl.java:1052) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl.removeRecord(ClusterConnectionImpl.java:1619) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionBridge.fail(ClusterConnectionBridge.java:436) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl.connectionFailed(BridgeImpl.java:699) ~[artemis-server-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:878) ~[artemis-core-client-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:804) ~[artemis-core-client-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:576) ~[artemis-core-client-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1417) ~[artemis-core-client-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98) ~[artemis-core-client-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:209) ~[artemis-core-client-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1182) ~[artemis-core-client-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57) ~[artemis-commons-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) ~[artemis-commons-2.35.0.jar:2.35.0]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) ~[artemis-commons-2.35.0.jar:2.35.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.35.0.jar:2.35.0]{noformat}
I think this logic could be improved to simply log when a binding can't be found in this context. I'll send a PR for this.
That said, this is not normal behavior, and I haven't been able to reproduce this in my local tests.
> The load balancing is not working correctly when several brokers are down
> -------------------------------------------------------------------------
>
> Key: ARTEMIS-5042
> URL: https://issues.apache.org/jira/browse/ARTEMIS-5042
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Balancer
> Affects Versions: 2.35.0
> Environment: Cluster Configuration broker.xml:
>
> <cluster-connections>
> <cluster-connection name="mccon">
> <connection-ttl>60000</connection-ttl>
> <call-timeout>5000</call-timeout>
> <retry-interval>500</retry-interval>
> <retry-interval-multiplier>1.0</retry-interval-multiplier>
> <max-retry-interval>5000</max-retry-interval>
> <initial-connect-attempts>-1</initial-connect-attempts>
> <reconnect-attempts>-1</reconnect-attempts>
> <forward-when-no-consumers>false</forward-when-no-consumers>
> <connector-ref>artemis</connector-ref>
> <message-load-balancing>ON_DEMAND</message-load-balancing>
> <max-hops>1</max-hops>
> <static-connectors>
> <connector-ref>node0</connector-ref>
> <connector-ref>node1</connector-ref>
> <connector-ref>node2</connector-ref>
>
> </static-connectors>
> </cluster-connection>
> </cluster-connections>
> Reporter: Juanjo Marin
> Assignee: Justin Bertram
> Priority: Major
> Attachments: 1NodoUp.PNG, 1nodeUp.PNG, 3NodesUp.PNG, 3Nodosup.PNG, ArtemisInternalWithMessage.PNG, EmptysQueue.PNG, QueueInternWithMessageNoConsumer.PNG, artemis.log, broker.xml
>
>
> The load balancing is not working correctly when several brokers are down. With the configuration we have in the cluster, we understand that this should not be happening. The load balancing should be smart and not send messages to the nodes that are down."
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@activemq.apache.org
For additional commands, e-mail: issues-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact