You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Jens De Temmerman (Jira)" <ji...@apache.org> on 2021/02/02 09:16:00 UTC
[jira] [Comment Edited] (ARTEMIS-3075) Scale down of
activemq.notifications causes issues
[ https://issues.apache.org/jira/browse/ARTEMIS-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276970#comment-17276970 ]
Jens De Temmerman edited comment on ARTEMIS-3075 at 2/2/21, 9:15 AM:
---------------------------------------------------------------------
Simply having a large amount of consumers on the server that is being shutdown seems to be enough. I believe it's a timing issue, so it's tricky to write a testcase for it that reproduces the issue. But I could reproduce it by creating 50 consumers on an instance, then shutting it down. It caused 13 messages to remain on the notification queue ("CONSUMER_CLOSED" messages) in one testrun.
I've created a testcase here: [https://github.com/jensdt/ARTEMIS-3075]
was (Author: jensdt-basf):
Simply having a large amount of consumers on the server that is being shutdown seems to be enough. I believe it's a timing issue, so it's tricky to write a testcase for it that reproduces the issue. But I could reproduce it by creating 50 consumers on an instance, then shutting it down. It cause 13 messages to remain on the notification queue ("CONSUMER_CLOSED" messages).
I've created a testcase here: https://github.com/jensdt/ARTEMIS-3075
> Scale down of activemq.notifications causes issues
> --------------------------------------------------
>
> Key: ARTEMIS-3075
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3075
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Affects Versions: 2.16.0
> Reporter: Jens De Temmerman
> Priority: Major
>
> Situation: a 2 server cluster, live-live, configured statically. Servers are configured to scale-down to each other when shut down.
> When the servers are live, they create a queue on each other's "activemq.notifications" address. This is done in ClusterConnectionBridge, I assume this is needed to get clustering to work properly.
> The queues are named "notif.6dca3cd4-5a61-11eb-8600-005056a1a158.ActiveMQServerImpl_serverUUID=903148b3-4455-11eb-bfed-005056a1a158" with serverUUID on server B pointing to server A and vice versa. (the UUID after "notif." is random).
> However, when server A is shut down, in some cases messages remain on this queue. This may be a timing issue, but regardless, when scaling down starts the ScaleDownHandler finds messages:
>
> {code:java}
> 2021-01-19 15:32:42,780 DEBUG [org.apache.activemq.artemis.core.server.impl.ScaleDownHandler] Scaling down address activemq.notifications
> 2021-01-19 15:32:42,781 DEBUG [org.apache.activemq.artemis.core.server.impl.ScaleDownHandler] Scaling down messages on address activemq.notifications
> 2021-01-19 15:32:42,781 DEBUG [org.apache.activemq.artemis.core.server.impl.ScaleDownHandler] Scaling down messages on address activemq.notifications / performing loop on queue QueueImpl[name=notif.6dca3cd4-5a61-11eb-8600-005056a1a158.ActiveMQServerImpl_serverUUID=903148b3-4455-11eb-bfed-005056a1a158, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=bbd13595-4473-11eb-aadd-005056a1b044], temp=true]@64be4624
> 2021-01-19 15:32:42,786 DEBUG [org.apache.activemq.artemis.core.server.impl.ScaleDownHandler] Reading message CoreMessage[messageID=796047343,durable=true,userID=null,priority=0, timestamp=Tue Jan 19 15:32:42 CET 2021,expiration=0, durable=true, address=activemq.notifications,size=1000,properties=TypedProperties[_AMQ_Distance=0,_AMQ_ConsumerCount=0,_AMQ_User=REDACTED,_AMQ_ROUTING_TYPE=0,_AMQ_SessionName=7b37ba58-5a5d-11eb-b854-005056bd7345,_AMQ_Address=REDACTED,_AMQ_RemoteAddress=/REDACTED:42310,_AMQ_NotifTimestamp=1611066762756,_AMQ_ClusterName=2f88994d-5d37-4976-afcd-b879fb80993abbd13595-4473-11eb-aadd-005056a1b044,_AMQ_RoutingName=2f88994d-5d37-4976-afcd-b879fb80993a,_AMQ_NotifType=CONSUMER_CLOSED,_AMQ_FilterString=NULL-value]]@187982968 from queue QueueImpl[name=notif.6dca3cd4-5a61-11eb-8600-005056a1a158.ActiveMQServerImpl_serverUUID=903148b3-4455-11eb-bfed-005056a1a158, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=bbd13595-4473-11eb-aadd-005056a1b044], temp=true]@64be4624
> {code}
> Of course, this queue doesn't exist on node B. But this causes this queue to be created on node B:
> {code}
> 2021-01-19 15:32:42,814 DEBUG [org.apache.activemq.artemis.core.server.impl.ScaleDownHandler] Failed to get queue ID, creating queue [addressName=activemq.notifications, queueName=notif.6dca3cd4-5a61-11eb-8600-005056a1a158.ActiveMQServerImpl_serverUUID=903148b3-4455-11eb-bfed-005056a1a158, routingType=MULTICAST, filter=_AMQ_Binding_Type<>2 AND _AMQ_NotifType IN ('SESSION_CREATED','BINDING_ADDED','BINDING_REMOVED','CONSUMER_CREATED','CONSUMER_CLOSED','PROPOSAL','PROPOSAL_RESPONSE','UNPROPOSAL') AND _AMQ_Distance<1 AND (((_AMQ_Address NOT LIKE 'activemq%') AND (_AMQ_Address NOT LIKE '$.artemis.internal.sf.%') AND (_AMQ_Address NOT LIKE 'activemq.management%'))) AND (_AMQ_NotifType = 'SESSION_CREATED' OR (_AMQ_Address NOT LIKE 'activemq.notifications%')), durable=false]
> {code}
> But on node B, there will never be any consumers on this queue. Since activemq.notifications is multicast, all messages sent to it are distributed to this queue as well, but never consumed nor purged (unless done manually).
> I'm not sure what the correct course of action is, but somehow this queue should not be considered for scale-down, as it creates a situation that requires manual intervention to fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)