[GitHub] [pulsar] 315157973 commented on a change in pull request #8418: support broker-level consume rate limiter

[GitBox <gi...@apache.org>]

315157973 commented on a change in pull request #8418:
URL: https://github.com/apache/pulsar/pull/8418#discussion_r515787156



##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java
##########
@@ -393,6 +397,16 @@ protected void readMoreEntries(Consumer consumer) {
             // active-cursor reads message from cache rather from bookkeeper (2) if topic has reached message-rate
             // threshold: then schedule the read after MESSAGE_RATE_BACKOFF_MS
             if (serviceConfig.isDispatchThrottlingOnNonBacklogConsumerEnabled() || !cursor.isActive()) {
+                //If it reaches the threshold of the entire Broker rate limit, try to read again later.
+                if (topic.getBrokerService().getBrokerDispatchRateLimiter() != null
+                        && topic.getBrokerService().getBrokerDispatchRateLimiter().isDispatchRateLimitingEnabled()
+                        && topic.getBrokerService().getBrokerDispatchRateLimiter().isConsumeRateExceeded()) {

Review comment:
       All current limiting in Pulsar will have this problem:
   We judge whether the threshold is exceeded before reading the entry; but increasing the count of msg and byte is after reading data, so there will be a time difference.
   Even if we use the precise Limiter, the count will not increase until the data is read. If you read concurrently in an instant, the threshold will still be exceeded.
   MultiConsumer will create multiple Consumers to consume each partition at the same time, which is considered as a concurrent read.
   
   If we want to be absolutely accurate, we may need to make Cursor judge whether it exceeds the threshold every time it reads a message. I think this is a huge loss in performance.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org