[GitHub] [pulsar] wangjialing218 commented on a change in pull request #8418: support broker-level consume rate limiter

[GitBox <gi...@apache.org>]

wangjialing218 commented on a change in pull request #8418:
URL: https://github.com/apache/pulsar/pull/8418#discussion_r515826034



##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java
##########
@@ -393,6 +397,16 @@ protected void readMoreEntries(Consumer consumer) {
             // active-cursor reads message from cache rather from bookkeeper (2) if topic has reached message-rate
             // threshold: then schedule the read after MESSAGE_RATE_BACKOFF_MS
             if (serviceConfig.isDispatchThrottlingOnNonBacklogConsumerEnabled() || !cursor.isActive()) {
+                //If it reaches the threshold of the entire Broker rate limit, try to read again later.
+                if (topic.getBrokerService().getBrokerDispatchRateLimiter() != null
+                        && topic.getBrokerService().getBrokerDispatchRateLimiter().isDispatchRateLimitingEnabled()
+                        && topic.getBrokerService().getBrokerDispatchRateLimiter().isConsumeRateExceeded()) {

Review comment:
       Yes, I also found current topic level `DispatchRateLimiter` take no effect when multi subscriptions are created for this topic to do concurrent read. But for broker level consume,  concurrent read happens all the time, which will make broker level rate limiter take no effect.
   I have changed the behaviour of `RateLimter`, increase the byte count after data is read, and minus threshold byte every second instead of reset count to 0. Thus the reading will stop for a while and the final broker level threshold could be limited as expect.
   I did not test the loss of performance in this way, and still can not solve the problem which cause large traffic of consume at the moment of starting the consumer, which is mentioned in #7720




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org