You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "Mooshua (via GitHub)" <gi...@apache.org> on 2025/08/27 22:15:43 UTC
[I] Randomize the WaitForExclusive queue [pulsar]
Mooshua opened a new issue, #24676:
URL: https://github.com/apache/pulsar/issues/24676
### Search before reporting
- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
Suppose we have three nodes each receiving updates for some resources to provide values to pulsar consumers. We want the load to be balanced between each of these three nodes evenly, in order to reduce the load on both the brokers and the nodes themselves. Basically, instead of using WaitForExclusive for *leader elections*, we use them for load balancing.
Currently, I assign each topic a primary node. When that node creates a producer for that topic, it uses a `ExclusiveWithFencing` access mode, and the two "standby" nodes for that topic establish `WaitForExclusive` producers.
Therefore, the connection process is:
- Iterate all watched resources
- For each resource we are a primary for, establish a `ExclusiveWithFence` producer for that topic
- For each resource we are a standby for, establish a `WaitForExclusive` producer for that topic
- When any of our producers are created, connect to the external resource update channel and stream updates into pulsar, effectively taking on the load for that resource
For my use case, I want nodes to be able to regularly drop out in order to perform rolling updates or other maintenance. **However, when they drop, their producer channels tend to go to *one* node, the first node to register WaitForExclusive access with the brokers.** This leaves one node running 2/3rds of the cluster's work, and the other node only carrying 1/3.
I'm fine with this sort of failure state, but I feel like this change would be a simple way to improve the failure scenario for use cases like mine.
### Solution
When a producer files `WaitForExclusive` access with the broker, and it is not immediately assigned exclusitivity, it should be put into a *random* spot in the queue, rather than the same spot every time. This would ensure that, should a primary node fail, it's work would be randomly distributed among the remaining nodes.
### Alternatives
One alternative is to sort the queue based on a producer-provided priority. This would allow producers to establish an order of which producer gets exclusivity when the exclusive producer closes. This exact functionality could then be implemented on the producer side by having each producer pick a random priority, creating a random order. This could also have use cases far beyond randomizing producer rebalancing.
For example, creating a priority queue could enable establishing a whole other cluster of fallover producers in a backup region, and de-prioritizing those so they only gain control when the entire "primary" cluster gets taken down.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Enhancement] Randomize the WaitForExclusive queue for producers [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #24676:
URL: https://github.com/apache/pulsar/issues/24676#issuecomment-3232588833
> I can absolutely open a PR for the "shuffle the waiting producer queue" option if that sounds like a good addition :)
>
> Q: Is the promotion order of producers ever explicitly defined anywhere, or is it an implementation detail? I combed the docs and I couldn't find anything.
`WaitForExclusive` mode was defined in https://github.com/apache/pulsar/blob/master/pip/pip-68.md .
I guess you are already aware of the code location where the queue is: https://github.com/apache/pulsar/blob/27df79edd0ffdf105c6b07220be90be96361ff07/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java#L839-L848
It seems to be a FIFO queue currently. It definitely makes sense to support producer provided priority and use a PriorityQueue. I guess it would be possible to retain the previous FIFO behavior by assigning the priorities in a way that it results in FIFO order if the priority isn't provided by the client for a producer.
Adding priority for producers raises the question whether there should also be an exclusive mode where a higher priority producer would always takeover a lower priority producer. Something like an ExclusivePriority mode.
btw. The solution (including rejected alternatives) would be documented in a PIP (Pulsar Improvement Proposal) document besides updating user facing documentation. The PIP process itself is fairly light weight and is explained in https://github.com/apache/pulsar/tree/master/pip#pulsar-improvement-proposal-pip .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Enhancement] Randomize the WaitForExclusive queue for producers [pulsar]
Posted by "Mooshua (via GitHub)" <gi...@apache.org>.
Mooshua commented on issue #24676:
URL: https://github.com/apache/pulsar/issues/24676#issuecomment-3234341893
> It seems to be a FIFO queue currently. It definitely makes sense to support producer provided priority and use a PriorityQueue. I guess it would be possible to retain the previous FIFO behavior by assigning the priorities in a way that it results in FIFO order if the priority isn't provided by the client for a producer.
One option is to sort by priority, *then* by add order. A default priority is chosen if a client doesn't specify. So, existing producers still see FIFO behavior, but an "enlightened" producer could choose to adopt priority.
> Adding priority for producers raises the question whether there should also be an exclusive mode where a higher priority producer would always takeover a lower priority producer. Something like an ExclusivePriority mode.
Hmm. I mean, this is already *somewhat* possible with `ExclusiveWithFence` so I don't see any obvious motivating examples. Eg, a producer could publish it's priority as part of it's messages, and a node with lower priority could pre-empt that producer by using `ExclusiveWithFence` and then publishing a higher priority.
> The solution (including rejected alternatives) would be documented in a PIP (Pulsar Improvement Proposal) document besides updating user facing documentation
Gotcha. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Enhancement] Randomize the WaitForExclusive queue for producers [pulsar]
Posted by "Mooshua (via GitHub)" <gi...@apache.org>.
Mooshua commented on issue #24676:
URL: https://github.com/apache/pulsar/issues/24676#issuecomment-3232463569
I can absolutely open a PR for the "shuffle the waiting producer queue" option if that sounds like a good addition :)
Q: Is the promotion order of producers ever explicitly defined anywhere, or is it an implementation detail? I combed the docs and I couldn't find anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Enhancement] Randomize the WaitForExclusive queue for producers [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #24676:
URL: https://github.com/apache/pulsar/issues/24676#issuecomment-3232088697
Thanks for the well written enhancement request @Mooshua! Would you be interested in contributing changes for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org