[GitHub] [spark] ivoson commented on pull request #37268: [SPARK-39853][CORE] Support stage level task resource schedule for standalone cluster when dynamic allocation disabled

[GitBox <gi...@apache.org>]

ivoson commented on PR #37268:
URL: https://github.com/apache/spark/pull/37268#issuecomment-1201323830

   > Can we please add more of a high level design/overview here?
   > 
   > Also I assume the intent here is to reuse executors since no dynamic allocation - please see #33941
   
   Hi @tgravescs , thanks for your feedback. You are right, the idea here is to reuse executors with `default resource profile` when dynamic allocation is disabled so that we can specify task resource profiles when dynamic allocation is off.
   
   This PR introduces a new special `ResourceProfile` (`TaskResourceProfile`) which describe different task resources, but share the same set of executors with default resource profile. Pretty similar to the concept of PR #33941, `TaskResourceProfile` is kind of special `ResourceProfile` which is compatible to default resource profile, so that we can support stage level task resource profiles when dynamic allocation is off.
   
   > Why is this only being added for standalone mode?
   It should be the same for Yarn and K8s cluster also. But I didn't test it in Yarn and K8s cluster right now, so I just enabled for standalone cluster in this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org