[GitHub] [pulsar] Jason918 opened a new issue #12651: [PIP] Add rate limiter to topic loading throttling.

[GitBox <gi...@apache.org>]

Jason918 opened a new issue #12651:
URL: https://github.com/apache/pulsar/issues/12651


   
   ## Motivation
   
   It's known that concurrent topic loading can create back pressure at zk and may increase zk latency,
   and Pulsar have introduced `maxConcurrentTopicLoadRequest` to limit concurrency of topic loading procedure.
   
   Currently we are running pulsar broker cluster with about 1 million topics on it, 
   and it's possible that 20 percent of our brokers could break down at the same time (by SLA of our infrastructure provider). 
   So when it happens, it means that 200K of topics needs to be loaded as soon as possible.
   
   
   Here is the problems:
   1. It's hard to determine the proper value for `maxConcurrentTopicLoadRequest`. For example, if we reduce this value to half, the loading speed dose not become half.
   2. By limiting the load concurrency, it's not easy to estimate the recover time when broker shutdown unexpectedly, and our SLA mostly depends on this in our case.
   
   ## Goal
   
   So I propose to add rate limit to the topic loading throttling.
   
   1. It's easy to choose proper value in perf test. It has about linear relationship with zk cpu usage.
   2. It's easy to estimate the max time of topic load costs.
   
   
   ## API Changes
   
   Add a config of `topicLoadRateLimit`, defining the max number of topics can be loaded in 1 second.
   
   ## Implementation
   
   The basic procedure is the same as throttling by concurrency.
   
   Add a field of type RateLimiter in BrokerService and try acquire a permit before `createPersistentTopic` in `BrokerService#loadOrCreatePersistentTopic`.
   
   
   ## Reject Alternatives
   
   No alternatives yet.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on issue #12651: [PIP] Add rate limiter to topic loading throttling.

[GitBox <gi...@apache.org>]

github-actions[bot] commented on issue #12651:
URL: https://github.com/apache/pulsar/issues/12651#issuecomment-1052937721


   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org