You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Thomas Graves (Jira)" <ji...@apache.org> on 2019/09/18 12:34:00 UTC

[jira] [Created] (SPARK-29149) Update YARN cluster manager For Stage Level Scheduling

Thomas Graves created SPARK-29149:
-------------------------------------

             Summary: Update YARN cluster manager For Stage Level Scheduling
                 Key: SPARK-29149
                 URL: https://issues.apache.org/jira/browse/SPARK-29149
             Project: Spark
          Issue Type: Bug
          Components: YARN
    Affects Versions: 3.0.0
            Reporter: Thomas Graves


For Stage Level Scheduling, we need to update the YARN allocator to handle requesting executors for multiple ResourceProfiles.
 * The container requests have to be updated to be based on a number of containers per ResourceProfile.  This is a larger change then you might expect because on YARN you can’t ask for different container sizes within the same YARN container priority.  So we will have to ask for containers at different priorities to be able to get different container sizes. Other YARN applications like Tez handle this now so it shouldn’t be a big deal just a matter of mapping stages to different priorities.
 * The allocation response from YARN has to match the containers to a resource profile.
 * We need to launch the container with additional parameters so the executor knows its resource profile to report back to the ExecutorMonitor.  

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org