You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Tobias Månsson (Jira)" <ji...@apache.org> on 2021/12/08 13:59:00 UTC
[jira] [Commented] (ARTEMIS-3602) Cluster connection not distribution messages
[ https://issues.apache.org/jira/browse/ARTEMIS-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17455252#comment-17455252 ]
Tobias Månsson commented on ARTEMIS-3602:
-----------------------------------------
Here are logs from a failed delivery
December 8th 2021, 13:47:59.234 artemis-r4xg7 2021-12-08 12:47:59,234 DEBUG [org.apache.activemq.artemis.core.server.impl.QueueMessageMetrics] QueuePendingMessageMetrics[queue=$.artemis.internal.sf.artemis.994daab0-51e4-11ec-bfd9-9ab74d93b535, name=pending] increment messageCount to 825: Reference[180276]:NON-RELIABLE:AMQPStandardMessage( [durable=false, messageID=180276, address=test.amqp.websocket.in, size=583, applicationProperties=\{msgType=test}, messageAnnotations=\{x-opt-ORIG-QUEUE=test-amqp-websocket-1, x-opt-ORIG-ADDRESS=test.amqp.websocket.out, x-opt-ORIG-MESSAGE-ID=180275}, properties=Properties\{messageId=null, userId=null, to='test.amqp.websocket.in', subject='null', replyTo='null', correlationId=null, contentType=null, contentEncoding=null, absoluteExpiryTime=Wed Dec 08 12:48:09 UTC 2021, creationTime=null, groupId='null', groupSequence=null, replyToGroupId='null'}, extraProperties = TypedProperties[_AMQ_ROUTE_TO$.artemis.internal.sf.artemis.994daab0-51e4-11ec-bfd9-9ab74d93b535=[0000 0000 0000 0087),bytesAsLongs(135],_AMQ_ORIG_ADDRESS=test.amqp.websocket.out,_AMQ_ORIG_MESSAGE_ID=180275,_AMQ_ORIG_QUEUE=test-amqp-websocket-1,_AMQ_AD=test.amqp.websocket.in]]
December 8th 2021, 13:47:59.234 artemis-r4xg7 2021-12-08 12:47:59,234 DEBUG [org.apache.activemq.artemis.core.server.impl.QueueImpl] Queue $.artemis.internal.sf.artemis.994daab0-51e4-11ec-bfd9-9ab74d93b535 doing deliver. messageReferences=824 with consumers=0
December 8th 2021, 13:47:59.233 artemis-r4xg7 2021-12-08 12:47:59,233 INFO [com.husqvarnagroup.dss.husqiot.broker.interceptor.MessageInterceptor] INCOMING interceptor accepted message with MessageAnnotations\{{x-opt-websocketConnections=hib:husqtest:KB9rqflVDoECJ0Q=}} and ApplicationProperties\{{msgType=test}} at test.amqp.websocket.out in 0 ms
December 8th 2021, 13:47:59.193 artemis-r4xg7 2021-12-08 12:47:59,192 DEBUG [org.apache.activemq.artemis.core.server.impl.QueueMessageMetrics] QueuePendingMessageMetrics[queue=test.amqp.websocket.in, name=delivering] decrement messageCount to 0: Reference[180273]:NON-RELIABLE:AMQPStandardMessage( [durable=false, messageID=180273, address=test.amqp.websocket.in, size=583, applicationProperties=\{msgType=test}, messageAnnotations=\{x-opt-ORIG-QUEUE=test-amqp-websocket-1, x-opt-ORIG-ADDRESS=test.amqp.websocket.out, x-opt-ORIG-MESSAGE-ID=180272}, properties=Properties\{messageId=null, userId=null, to='test.amqp.websocket.in', subject='null', replyTo='null', correlationId=null, contentType=null, contentEncoding=null, absoluteExpiryTime=Wed Dec 08 12:48:09 UTC 2021, creationTime=null, groupId='null', groupSequence=null, replyToGroupId='null'}, extraProperties = TypedProperties[_AMQ_ORIG_ADDRESS=test.amqp.websocket.out,_AMQ_ORIG_MESSAGE_ID=180272,_AMQ_ORIG_QUEUE=test-amqp-websocket-1,_AMQ_AD=test.amqp.websocket.in]]
December 8th 2021, 13:47:59.190 artemis-r4xg7 2021-12-08 12:47:59,190 DEBUG [org.apache.activemq.artemis.core.server.impl.QueueMessageMetrics] QueuePendingMessageMetrics[queue=test.amqp.websocket.in, name=pending] decrement messageCount to 0: Reference[180273]:NON-RELIABLE:AMQPStandardMessage( [durable=false, messageID=180273, address=test.amqp.websocket.in, size=583, applicationProperties=\{msgType=test}, messageAnnotations=\{x-opt-ORIG-QUEUE=test-amqp-websocket-1, x-opt-ORIG-ADDRESS=test.amqp.websocket.out, x-opt-ORIG-MESSAGE-ID=180272}, properties=Properties\{messageId=null, userId=null, to='test.amqp.websocket.in', subject='null', replyTo='null', correlationId=null, contentType=null, contentEncoding=null, absoluteExpiryTime=Wed Dec 08 12:48:09 UTC 2021, creationTime=null, groupId='null', groupSequence=null, replyToGroupId='null'}, extraProperties = TypedProperties[_AMQ_ORIG_ADDRESS=test.amqp.websocket.out,_AMQ_ORIG_MESSAGE_ID=180272,_AMQ_ORIG_QUEUE=test-amqp-websocket-1,_AMQ_AD=test.amqp.websocket.in]]
December 8th 2021, 13:47:59.190 artemis-r4xg7 2021-12-08 12:47:59,190 DEBUG [org.apache.activemq.artemis.core.server.impl.QueueMessageMetrics] QueuePendingMessageMetrics[queue=test.amqp.websocket.in, name=delivering] increment messageCount to 1: Reference[180273]:NON-RELIABLE:AMQPStandardMessage( [durable=false, messageID=180273, address=test.amqp.websocket.in, size=583, applicationProperties=\{msgType=test}, messageAnnotations=\{x-opt-ORIG-QUEUE=test-amqp-websocket-1, x-opt-ORIG-ADDRESS=test.amqp.websocket.out, x-opt-ORIG-MESSAGE-ID=180272}, properties=Properties\{messageId=null, userId=null, to='test.amqp.websocket.in', subject='null', replyTo='null', correlationId=null, contentType=null, contentEncoding=null, absoluteExpiryTime=Wed Dec 08 12:48:09 UTC 2021, creationTime=null, groupId='null', groupSequence=null, replyToGroupId='null'}, extraProperties = TypedProperties[_AMQ_ORIG_ADDRESS=test.amqp.websocket.out,_AMQ_ORIG_MESSAGE_ID=180272,_AMQ_ORIG_QUEUE=test-amqp-websocket-1,_AMQ_AD=test.amqp.websocket.in]]
December 8th 2021, 13:47:59.190 artemis-r4xg7 2021-12-08 12:47:59,190 INFO [com.husqvarnagroup.dss.husqiot.broker.interceptor.MessageInterceptor] OUTGOING interceptor accepted message with MessageAnnotations\{{x-opt-ORIG-QUEUE=test-amqp-websocket-1, x-opt-ORIG-ADDRESS=test.amqp.websocket.out, x-opt-ORIG-MESSAGE-ID=180272}} and ApplicationProperties\{{msgType=test}} at test.amqp.websocket.in in 0 ms
December 8th 2021, 13:47:59.189 artemis-r4xg7 2021-12-08 12:47:59,189 INFO [com.husqvarnagroup.dss.husqiot.broker.interceptor.MessageInterceptor] INCOMING interceptor accepted message with MessageAnnotations\{{x-opt-websocketConnections=hib:husqtest:KB9rqflVDoECJ0Q=}} and ApplicationProperties\{{msgType=test}} at test.amqp.websocket.out in 1 ms
December 8th 2021, 13:47:59.189 artemis-r4xg7 2021-12-08 12:47:59,189 DEBUG [org.apache.activemq.artemis.core.server.impl.QueueImpl] Queue test.amqp.websocket.in doing deliver. messageReferences=0 with consumers=1
December 8th 2021, 13:47:59.189 artemis-r4xg7 2021-12-08 12:47:59,189 DEBUG [org.apache.activemq.artemis.core.server.impl.QueueMessageMetrics] QueuePendingMessageMetrics[queue=test.amqp.websocket.in, name=pending] increment messageCount to 1: Reference[180273]:NON-RELIABLE:AMQPStandardMessage( [durable=false, messageID=180273, address=test.amqp.websocket.in, size=583, applicationProperties=\{msgType=test}, messageAnnotations=\{x-opt-ORIG-QUEUE=test-amqp-websocket-1, x-opt-ORIG-ADDRESS=test.amqp.websocket.out, x-opt-ORIG-MESSAGE-ID=180272}, properties=Properties\{messageId=null, userId=null, to='test.amqp.websocket.in', subject='null', replyTo='null', correlationId=null, contentType=null, contentEncoding=null, absoluteExpiryTime=Wed Dec 08 12:48:09 UTC 2021, creationTime=null, groupId='null', groupSequence=null, replyToGroupId='null'}, extraProperties = TypedProperties[_AMQ_ORIG_ADDRESS=test.amqp.websocket.out,_AMQ_ORIG_MESSAGE_ID=180272,_AMQ_ORIG_QUEUE=test-amqp-websocket-1,_AMQ_AD=test.amqp.websocket.in]]
> Cluster connection not distribution messages
> --------------------------------------------
>
> Key: ARTEMIS-3602
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3602
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Affects Versions: 2.18.0
> Reporter: Tobias Månsson
> Priority: Major
>
> The cluster is set up as a ON_DEMAND cluster with two nodes, were only one node is exposed to the external load-balancer and receiving new connections. This creates a online failover setup by switching over new connections to the other node, before taking down the current for maintenance.
> During one deployment, the active broker node was failed over without shutting down the passive node. This left the current connection on the passive node, while new connections were created against the now active one.
> If message were sent to the now passive node, they were synchronized correctly to both nodes and to all connections. But if the messages were sent to then now active node (the intended behavior), they were not sent to clients currently connected to the now passive node.
> The logs and message counters confirm this, but no cause has been found. Since the cluster synchronization work one way, but not the other, I don't know where to search for errors. I've enabled debug logs on both servers, but don't see any issues. The messages are committed to the correct queues, but nothing is received on the other node in the failure scenario.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)