You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/06/23 10:10:08 UTC
[GitHub] [pulsar] anvinjain opened a new issue #11044: Dispatch rate discrepancies when byte rates used
anvinjain opened a new issue #11044:
URL: https://github.com/apache/pulsar/issues/11044
**Describe the bug**
When using byte rates, we see that dispatch rates are often not respected (regardless of being a namespace or topic policy)
**To Reproduce**
This happens when dispatch rates are set on the lower side, say, 100KB/s while dispatcherMaxReadSizeBytes=5MB and dispatcherMaxReadBatchSize=100 which are defaults
On debugging, we discovered that dispatcherMaxReadSizeBytes and dispatcherMaxReadBatchSize have an impact on the minimum dispatch rate which we see. Here although the settings have word "max" in them, when it comes to reading from bookkeeper (lagging consumers), these control the minimum data read (in terms of entries or bytes) when byte rates are used instead of message based limits in dispatch.
**Expected behavior**
Expected that dispatch rate = 100KB/s for a subscription, but saw that it was 5MB/s.
**Desktop (please complete the following information):**
- OS: Debian 10
- Pulsar 2.7.1
**Additional context**
In the code, I see that messagesToRead is initialized with readBatchSize and only reduced if message based rate limiting has been used: https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java#L397
When byte rate limiting is used, the first read can be much larger than the configured limit (say 5MB because of dispatcherMaxReadBatchSize vs limit of 1MB as configured in dispatch rate of namespace). Post the first read for that second, rate limiter permits gets exhausted rightly so but the user effectively sees 5MB/s dispatch. On the start of the next second again this happens where the first read issued to bookkeeper is larger than the overall configured limit and it exhausts the permits but effective rate/s remains to be 5MB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #11044: Dispatch rate discrepancies when byte rates used
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #11044:
URL: https://github.com/apache/pulsar/issues/11044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] anvinjain edited a comment on issue #11044: Dispatch rate discrepancies when byte rates used
Posted by GitBox <gi...@apache.org>.
anvinjain edited a comment on issue #11044:
URL: https://github.com/apache/pulsar/issues/11044#issuecomment-866709634
Continuing on the exploration, maybe this is as simple as changing https://github.com/apache/pulsar/blob/a070c3363d007198f5dee4d3d70d075ba6a1c8df/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java#L343 from serviceConfig.getDispatcherMaxReadSizeBytes() to Math.min(serviceConfig.getDispatcherMaxReadSizeBytes(), dispatch-byte-rate-limit)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on issue #11044: Dispatch rate discrepancies when byte rates used
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on issue #11044:
URL: https://github.com/apache/pulsar/issues/11044#issuecomment-867753183
hi @anvinjain , now only dispatch-rate-limit-by-msg-number can cumulate messagesToRead https://github.com/apache/pulsar/blob/0be9d1373f0077286812f2636acf451ee3f10e4b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java#L427 if want dispatch-rate-limit-by-byte precise, when cumulate messagesToRead by the dispatch-rate-limit-by-byte. I think it is a good way to fix it. Do you think this is good？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #11044: Dispatch rate discrepancies when byte rates used
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #11044:
URL: https://github.com/apache/pulsar/issues/11044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] anvinjain commented on issue #11044: Dispatch rate discrepancies when byte rates used
Posted by GitBox <gi...@apache.org>.
anvinjain commented on issue #11044:
URL: https://github.com/apache/pulsar/issues/11044#issuecomment-866709447
One way to fix this could to be penalize subsequent rate period too if we have gone over the limit in one of the period (in this case period = 1 second).
Another way would be to calculate messagesToRead proportionally according to byte limit. In fact this is even done here: https://github.com/apache/pulsar/blob/b9db969d6bc3e2d18ee35e84629c6634560fb622/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java#L3004 but here maxSizeBytes is coming from dispatcherMaxReadSizeBytes and not from the dispatch limit which user has configured.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] anvinjain commented on issue #11044: Dispatch rate discrepancies when byte rates used
Posted by GitBox <gi...@apache.org>.
anvinjain commented on issue #11044:
URL: https://github.com/apache/pulsar/issues/11044#issuecomment-866709634
Continuing on the exploration, maybe this is as simple as changing https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java#L343 from serviceConfig.getDispatcherMaxReadSizeBytes() to Math.min(serviceConfig.getDispatcherMaxReadSizeBytes(), dispatch-byte-rate-limit)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wangjialing218 commented on issue #11044: Dispatch rate discrepancies when byte rates used
Posted by GitBox <gi...@apache.org>.
wangjialing218 commented on issue #11044:
URL: https://github.com/apache/pulsar/issues/11044#issuecomment-872728506
@anvinjain This issue could be fixed in #8611 by this way, which already merged in 2.8.0. You can have a try.
> One way to fix this could to be penalize subsequent rate period too if we have gone over the limit in one of the period (in this case period = 1 second).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org