[GitHub] [spark] xwu99 commented on pull request #33941: [WIP][SPARK-36699][Core] Reuse compatible executors for stage-level scheduling

[GitBox <gi...@apache.org>]

xwu99 commented on pull request #33941:
URL: https://github.com/apache/spark/pull/33941#issuecomment-984263614


   Busy for a while, back to this topic. I wil find some time to address code comments above. 
   
   For reuse policy, there are two options in my mind right now: 
   1) **strict match**: only reuse executors with exact same resources (including all 3rd party resources), there is no resource waste but less user flexibility
   2) **reuse larger executor**: if there is a larger executor which has resources larger than or equal to current requirements, eg. if you define less memory in the new stage then you can reuse previous executor with larger memory. or if you define new stage with no GPU, you can reuse previous executor with GPU. in both cases, new stage has less resource requrements. But in this policy user should know there is some resource waste. They need to tradeoff reuse executor or create new ones.
   
   Do we also allow user to select reuse policy at stage level?
   
   I am not sure if all policies can be used in real-world scenerio. How about we implement some policy first to get things working for some scenerio and to leave policies options open to add new policy in the future? 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org