You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/06/27 12:27:07 UTC
[GitHub] [pulsar] codelipenghui opened a new pull request, #16241: [improve][broker] Reduce the re-schedule message read operation for PersistentDispatcherMultipleConsumers
codelipenghui opened a new pull request, #16241:
URL: https://github.com/apache/pulsar/pull/16241
### Motivation
Fix the CPU consumption while having many consumers (> 100k) and enabled
dispatch rate limit.
[broker_perf.html.txt](https://github.com/apache/pulsar/files/8991916/broker_perf.html.txt)
### Modification
- Added `isRescheduleReadInProgress` to ensure the dispatcher only has one pending re-schedule read task at a time.
- Added DEBUG log for the re-schedule read operation
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (no)
- The public API: (no)
- The schema: (no)
- The default values of configurations: (no)
- The wire protocol: (no)
- The rest endpoints: (no)
- The admin cli options: (no)
- Anything that affects deployment: (no)
### Documentation
Check the box below or label this PR directly.
Need to update docs?
- [ ] `doc-required`
(Your PR needs to update docs and you will update later)
- [x] `doc-not-needed`
(Please explain why)
- [ ] `doc`
(Your PR contains doc changes)
- [ ] `doc-complete`
(Docs have been already added)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on a diff in pull request #16241: [improve][broker] Reduce the re-schedule message read operation for PersistentDispatcherMultipleConsumers
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on code in PR #16241:
URL: https://github.com/apache/pulsar/pull/16241#discussion_r907931428
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java:
##########
@@ -291,8 +294,17 @@ public synchronized void readMoreEntries() {
@Override
protected void reScheduleRead() {
- topic.getBrokerService().executor().schedule(() -> readMoreEntries(), MESSAGE_RATE_BACKOFF_MS,
- TimeUnit.MILLISECONDS);
+ if (isRescheduleReadInProgress.compareAndSet(false, true)) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] [{}] Reschedule message read in {} ms", topic.getName(), name, MESSAGE_RATE_BACKOFF_MS);
+ }
+ topic.getBrokerService().executor().schedule(
Review Comment:
We may add the thread pool metric to get the queue length of each thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a diff in pull request #16241: [improve][broker] Reduce the re-schedule message read operation for PersistentDispatcherMultipleConsumers
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on code in PR #16241:
URL: https://github.com/apache/pulsar/pull/16241#discussion_r907946465
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java:
##########
@@ -291,8 +294,17 @@ public synchronized void readMoreEntries() {
@Override
protected void reScheduleRead() {
- topic.getBrokerService().executor().schedule(() -> readMoreEntries(), MESSAGE_RATE_BACKOFF_MS,
- TimeUnit.MILLISECONDS);
+ if (isRescheduleReadInProgress.compareAndSet(false, true)) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] [{}] Reschedule message read in {} ms", topic.getName(), name, MESSAGE_RATE_BACKOFF_MS);
+ }
+ topic.getBrokerService().executor().schedule(
Review Comment:
Yes. looks like currently we only have web executor metrics. @tjiuming Could you please help take a look at this part?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd commented on pull request #16241: [improve][broker] Reduce the re-schedule message read operation for PersistentDispatcherMultipleConsumers
Posted by GitBox <gi...@apache.org>.
shibd commented on PR #16241:
URL: https://github.com/apache/pulsar/pull/16241#issuecomment-1168170885
@codelipenghui How this graph of CPU usage is generated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #16241: [improve][broker] Reduce the re-schedule message read operation for PersistentDispatcherMultipleConsumers
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on PR #16241:
URL: https://github.com/apache/pulsar/pull/16241#issuecomment-1168176174
@shibd https://github.com/jvm-profiling-tools/async-profiler
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #16241: [improve][broker] Reduce the re-schedule message read operation for PersistentDispatcherMultipleConsumers
Posted by GitBox <gi...@apache.org>.
codelipenghui merged PR #16241:
URL: https://github.com/apache/pulsar/pull/16241
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org