[GitHub] [incubator-yunikorn-core] wangdatan commented on pull request #173: [YUNIKORN-231] Add ability to update pod condition based on accurate scheduling state

[GitBox <gi...@apache.org>]

wangdatan commented on pull request #173:
URL: https://github.com/apache/incubator-yunikorn-core/pull/173#issuecomment-654513909


   @wilfred-s I think what you suggested make sense, and (if you remember) that is why initially in YuniKorn core implementation, we have logic to get "allocatable resource request" first (which consider fairness for the "potential allocated" resource).  
   
   If we want to achieve a close-to-truth outstanding resource request sorting which covers point you suggested, it will be a major effort which will take months and ideally it needs to exchange information with autoscaler. 
   
   My suggestion is to get this done in phases, for phase #1, I think it is relatively safe to assume the following thing: 
   
   **Resource requests are relatively homogeneous**: which means if new node added for app1's total 200 GB pending mem, even if app1 somehow cannot consume the new node resource immediately (maybe caused by FIFO order). there will be some other pending pods can pick it up and use it.  
   
   If we agree with the above assumption, Weiwei's patch should be a good starting point for phase#1. If we want to do it better, here're follow-ups we can do:  
   
   - P1: Prioritize daemonset/service PODs for queues which are under utilized. Daemonset/service pods are likely to be heterogeneous and requires anti-affinity, etc.
   - P1: Do the algorithm in two passes: first pass will select queue which has usage under guaranteed capacity (usage ratio < 1.0), and fill them to >=1. second pass will fill queues to maximum capacity.  
   - P2: Sort queues/apps based on the possible resource usage. 
   
   Thoughts? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org