You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Thomas Graves (Jira)" <ji...@apache.org> on 2019/10/31 20:00:03 UTC
[jira] [Comment Edited] (SPARK-29415) Stage Level Sched: Add base
ResourceProfile and Request classes
[ https://issues.apache.org/jira/browse/SPARK-29415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16964360#comment-16964360 ]
Thomas Graves edited comment on SPARK-29415 at 10/31/19 7:59 PM:
-----------------------------------------------------------------
More details:
TaskResourceRequest - this supports taking a resourceName and an amount (Double for fractional resources). It only supports cpus (spark.task.cpus) and accelerator resource types (spark.*.resource.[resourceName].*. So user can specify cpus and resources like GPU's and FPGAS. The accerator type resources match what we already have for configs in the acceralator aware scheduling. [https://github.com/apache/spark/blob/master/docs/configuration.md#custom-resource-scheduling-and-configuration-overview]
ExecutorResourceRequest - this supports specifying the requirements for the executors. It supports all the configs needed for accelerator aware scheduling - {{spark.\{executor/driver}.resource.\{resourceName}.\{amount, discoveryScript, vendor} as well as heap memory, overhead memory, pyspark memory, and cores. In order to support memory types we added a "units" parameter into ExecutorResourceRequest. the other parameters resourceName, vendor, discoveryScript, amount all match the accelerator aware scheduling parameters.}}
ResourceProfile - this class takes in the executor and task requirement and holds them to be used by other components. For instance, we have to pass the executor resources into the cluster managers so it can ask for the proper containers. The requests have to also be passed into the executors when launched so they use the correct discovery Script. The task requirements are used by the scheduler to assign tasks to proper containers. {{ We also have a ResourceProfile object that has an accessor to get the default ResourceProfile. This is the profile generated from the configs the user passes in when the spark application is submitted. So it will have --executor-cores, memory, overhead memory, pyspark memory, accelerator resources the user all specified via --confs or properties file on submit.}}
was (Author: tgraves):
More details:
TaskResourceRequest - this supports taking a resourceName and an amount (Double for fractional resources). It only supports cpus (spark.task.cpus) and accelerator resource types (spark.*.resource.[resourceName].*. So user can specify cpus and resources like GPU's and FPGAS. The accerator type resources match what we already have for configs in the acceralator aware scheduling. [https://github.com/apache/spark/blob/master/docs/configuration.md#custom-resource-scheduling-and-configuration-overview]
ExecutorResourceRequest - this supports specifying the requirements for the executors. It supports all the configs needed for accelerator aware scheduling - {{spark.\{executor/driver}.resource.\{resourceName}.\{amount, discoveryScript, vendor} as well as heap memory, overhead memory, pyspark memory, and cores. In order to support memory types we added a "units" parameter into ExecutorResourceRequest. the other parameters resourceName, vendor, discoveryScript, amount all match the accelerator aware scheduling parameters.}}
{{}}
ResourceProfile - this class takes in the executor and task requirement and holds them to be used by other components. For instance, we have to pass the executor resources into the cluster managers so it can ask for the proper containers. The requests have to also be passed into the executors when launched so they use the correct discovery Script. The task requirements are used by the scheduler to assign tasks to proper containers. {{ We also have a ResourceProfile object that has an accessor to get the default ResourceProfile. This is the profile generated from the configs the user passes in when the spark application is submitted. So it will have --executor-cores, memory, overhead memory, pyspark memory, accelerator resources the user all specified via --confs or properties file on submit.}}
> Stage Level Sched: Add base ResourceProfile and Request classes
> ---------------------------------------------------------------
>
> Key: SPARK-29415
> URL: https://issues.apache.org/jira/browse/SPARK-29415
> Project: Spark
> Issue Type: Story
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Major
>
> this is just to add initial ResourceProfile, ExecutorResourceRequest and taskResourceRequest classes that are used by the other parts of the code.
> Initially we will have them private until we have other pieces in place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org