You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "vvivekiyer (via GitHub)" <gi...@apache.org> on 2023/09/28 18:02:00 UTC

[GitHub] [pinot] vvivekiyer commented on pull request #11699: Make ResourceLimitPolicy functions public to be used from scheduler plugins.

vvivekiyer commented on PR #11699:
URL: https://github.com/apache/pinot/pull/11699#issuecomment-1739782509

   @Jackie-Jiang 
   We are not using the PriorityBasedSchedulers. Just curious - can you point me to the race condition?
   
   About this change -  We have a use case in Linkedin where we want to query production tables using adhoc channels (tools, scripts, reports, etc). Note that these adhoc queries are not from the site-facing applications but still need to be executed on production tables with some reasonably degraded latency. 
   
   So we've built an internal scheduler (short term) that caps the amount of CPU resources given to these adhoc workloads while still letting the regular production workloads perform as usual. This is just a short term solution.
   
   For the long term, I'm planning to leverage the above idea to build a long-term policy based workload management solution. I'm writing a PEP document for the overall effort and I'll share it with OSS soon (week or two).  
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org