You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2023/12/06 14:21:00 UTC
[jira] [Work logged] (ARTEMIS-4527) Redistributor race when consumerCount reaches 0 in cluster
[ https://issues.apache.org/jira/browse/ARTEMIS-4527?focusedWorklogId=894308&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-894308 ]
ASF GitHub Bot logged work on ARTEMIS-4527:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 06/Dec/23 14:20
Start Date: 06/Dec/23 14:20
Worklog Time Spent: 10m
Work Description: AntonRoskvist opened a new pull request, #4705:
URL: https://github.com/apache/activemq-artemis/pull/4705
…ster
This is a very rare bug but when triggered, messages in the queue with 0 consumers will have the redistributors loop messages between some or all brokers in a cluster as fast as they can manage, until either some system resource or the clusterBridges producerFlowControl is reached. Will keep happening until consumers are added or cluster bridges are restarted.
I don't have a test for this but instead added a reproducer that works with a considerable amount of tweaks. Comments in the reproducer explains how to run it. The reproducer is _not_ a valid or reasonable use case... it builds on some unrelated work I did that accidentally triggered this. I have seen this multiple times in a production environment over the course of several years though, I've just been unable to reproduce it outside of production before accidentally stumbling on it recently.
Problem occurs when CONSUMER_CREATED notification arrive before the BINDING_ADDED notification.
When that happens the consumerCount for RemoteBinding is incorrect (something like 1-2 lower than actual consumerCount value).
Then when consumers disconnect, all are registered properly and RemoteBinding gets a negative consumerCount. The `isHighAcceptPriority` used by the redistributor checks for consumerCount == 0 but since count is now negative it returns as a valid destination.
Fix adds synchronization on the postoffice when processing createConsumer so then the previously issued addBinding for sure is done before continuing.
I also added double safety in the RemoteQueueBinding by not lowering consumerCount below 0 and also checking for consumerCount <= 0 instead of consumerCount == 0, though neither of these should really be necessary if the cluster notifications always arrive in the correct order.
If anyone can figure out a consistent way to trigger this issue I'd be happy to add it. Regardless, if the changes look good otherwise I think the reproducer should be removed rather than merged, leaving it here for verification purposes.
One final though is that perhaps all of the sort of create/add/remove operations in the `org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler` should be synchronized?
Something building on the current pattern of:
`onMessagePacket()`
```
switch
fast1:
fast1Stuff();
fast2:
fast2Stuff();
default:
slow()
slow:
switch
slow1:
slow1Stuff();
slow2:
slow2Stuff();
default:
synchronizedStuff()
synchronizedStuff:
switch
...
...
```
Issue Time Tracking
-------------------
Worklog Id: (was: 894308)
Remaining Estimate: 0h
Time Spent: 10m
> Redistributor race when consumerCount reaches 0 in cluster
> ----------------------------------------------------------
>
> Key: ARTEMIS-4527
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4527
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Reporter: Anton Roskvist
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is a very rare bug caused by cluster notifications arriving in the wrong order in some very specific circumstances
--
This message was sent by Atlassian Jira
(v8.20.10#820010)