[GitHub] rdhabalia opened a new pull request #1005: Load-balancer consider max-topic threshold on broker-ownership selection

[GitBox <gi...@apache.org>]

rdhabalia opened a new pull request #1005: Load-balancer consider max-topic threshold on broker-ownership selection
URL: https://github.com/apache/incubator-pulsar/pull/1005
 
 
   ### Motivation
   
   Right now, load-manager doesn't consider number of topics owned by a broker while selecting bundle ownership and it overloads few of the brokers with large number of brokers.
   eg:
   if cluster has 2 brokers: b1 and b2. where b2 owns few bundles and traffic and b1 not. Now, if load-manager receives lookup requests for 10K different bundles at the same time then 
   - load-manager has data that b2 is having higher load than b1 
   - but load-manager doesn't consider number of topics in those bundles and as b1 has lower load than b2 so, all 10K bundles will be assigned to broker b1.
   
   Now, if broker owns topic higher than certain number (eg: 60K) then it requires to allocate fix set of objects (eg: rate, locks, stats, producers, consumers) per topic which majorly impacts broker GC so, load-manager should consider max-topics count while selecting broker.
   
   Note: load-manager has load-shedding task but sometimes we want broker to prevent bundle unloading so, load-manager should do this work while selecting broker.
   
   ### Modifications
   
   - store number of topics under bundle-resource quota
   - consider number of topics for a bundle while selecting broker
   
   ### Result
   
   Load-manager should not overload broker with topics higher than max-limit.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services