[GitHub] [pinot] walterddr commented on pull request #10711: [multistage][cleanup] make leaf stage also registered with OpChainSchedulerService

["walterddr (via GitHub)" <gi...@apache.org>]

walterddr commented on PR #10711:
URL: https://github.com/apache/pinot/pull/10711#issuecomment-1533459715

   putting it back to draft. there's a fairness issue with this approach. 
   
   1. the leaf-stage execution is non-yielding thus it will always be in the ready queue upon register;
   2. the intermediate-stage execution is yielding thus they will be yielding at register unless data is already available for this stage; once yield it will be put to the back of the queue
   3. each time intermediate-stage need to fight for round-robin scheduler to clear all the previous leaf-stage executions coming in before intermediate stage can execute.
   
   This creates a fairness issue b/c not all threads are executing [collaborative multithreaded](https://en.wikipedia.org/wiki/Cooperative_multitasking) -- thus the "bad guys" always gets the priority. 
   
   possible redesign
   - implement the PiplineBreakerOperator in this PR so that PiplineBreakerOperator will actually yield leaf-stage operator until the leaf-stage execution finishes
       - this will also make that `serverExecutor.execute()` to be run on QueryRunner threadpool. which is different from the QueryWorker (for leaf) and QueryIntermediate (for intermediate stage)
   - use 2 threadpool for scheduler service
   - use priority queue
   
   CC @xiangfu0 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org