You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Thomas Graves (Jira)" <ji...@apache.org> on 2020/11/13 18:18:00 UTC

[jira] [Created] (SPARK-33447) Stage level scheduling, allow specifying other spark configs via ResourceProfile

Thomas Graves created SPARK-33447:
-------------------------------------

             Summary: Stage level scheduling, allow specifying other spark configs via ResourceProfile
                 Key: SPARK-33447
                 URL: https://issues.apache.org/jira/browse/SPARK-33447
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 3.1.0
            Reporter: Thomas Graves


With the addition of stage level scheduling and ResourceProfiles, we only allow certain resources to specified currently.  There are many other configs that users may want to change between changes. We should look at adding in perhaps SparkConf settings.

One very specific one that was brought up in review was ability to change yarn queue (if that is even possible in yarn) between stages because someone might want ot use one queue for ETL and a separate queue for ML where there are nodes with GPU.  Or perhaps node labels separately.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org