You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "poorbarcode (via GitHub)" <gi...@apache.org> on 2026/09/21 16:36:07 UTC
[PR] [fix][broker] Bound classic Key_Shared dispatcher replay queue look-ahead [pulsar]
poorbarcode opened a new pull request, #26677:
URL: https://github.com/apache/pulsar/pull/26677
### Motivation
In the classic Key_Shared dispatcher, messages are assigned to consumers according to their sticky-key hash. If a consumer becomes slow and exhausts its available permits, messages mapped to that consumer cannot be dispatched. To keep other consumers making progress, the dispatcher continues normal reads and places these undispatchable entries into the in-memory replay queue.
This can cause broker OOM when all of the following conditions hold:
- There were `12` consumers
- `consumer#xzqg9` was slow; other consumers have consumed all messages that they can.
- The dispatcher keeps reading entries and pushing messages(which should be consumed by `consumer#xzqg9`) into the redeliver queue since `consumer#xzqg9`'s permits is `0`.
- Eventually, the queue is too large, causing broker OOM.
### Modifications
Let classic Key_Shared dispatcher support the configuration `keySharedLookAheadMsgInReplayThresholdPerSubscription`
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Bound classic Key_Shared dispatcher replay queue look-ahead [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #26677:
URL: https://github.com/apache/pulsar/pull/26677#issuecomment-5775512295
@poorbarcode I'm just wondering why wouldn't users just use the current dispatcher instead of the legacy one. The current PIP-379 has been available for almost 2 years and it has been battle tested by the majority of all Pulsar users in the community. Most users don't disable PIP-379. After 4.0.x there has been a very critical fixes for PIP-379 implementation which have been quickly fixed. At the same time, the legacy dispatcher has a lot of known issues that remain unfixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Bound classic Key_Shared dispatcher replay queue look-ahead [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode merged PR #26677:
URL: https://github.com/apache/pulsar/pull/26677
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org