You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Thomas Graves (Jira)" <ji...@apache.org> on 2019/09/18 12:36:00 UTC
[jira] [Updated] (SPARK-29149) Update YARN cluster manager For
Stage Level Scheduling
[ https://issues.apache.org/jira/browse/SPARK-29149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated SPARK-29149:
----------------------------------
Issue Type: Story (was: Bug)
> Update YARN cluster manager For Stage Level Scheduling
> ------------------------------------------------------
>
> Key: SPARK-29149
> URL: https://issues.apache.org/jira/browse/SPARK-29149
> Project: Spark
> Issue Type: Story
> Components: YARN
> Affects Versions: 3.0.0
> Reporter: Thomas Graves
> Priority: Major
>
> For Stage Level Scheduling, we need to update the YARN allocator to handle requesting executors for multiple ResourceProfiles.
> * The container requests have to be updated to be based on a number of containers per ResourceProfile. This is a larger change then you might expect because on YARN you can’t ask for different container sizes within the same YARN container priority. So we will have to ask for containers at different priorities to be able to get different container sizes. Other YARN applications like Tez handle this now so it shouldn’t be a big deal just a matter of mapping stages to different priorities.
> * The allocation response from YARN has to match the containers to a resource profile.
> * We need to launch the container with additional parameters so the executor knows its resource profile to report back to the ExecutorMonitor.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org