You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2021/11/05 21:20:00 UTC
[jira] [Resolved] (SPARK-37208) Support mapping Spark gpu/fpga
resource types to custom YARN resource type
[ https://issues.apache.org/jira/browse/SPARK-37208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun resolved SPARK-37208.
-----------------------------------
Fix Version/s: 3.2.1
3.3.0
Resolution: Fixed
Issue resolved by pull request 34485
[https://github.com/apache/spark/pull/34485]
> Support mapping Spark gpu/fpga resource types to custom YARN resource type
> --------------------------------------------------------------------------
>
> Key: SPARK-37208
> URL: https://issues.apache.org/jira/browse/SPARK-37208
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 3.0.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Major
> Fix For: 3.3.0, 3.2.1
>
>
> Currently Spark supports gpu/fpga resource scheduling and specifically on YARN it knows how to map gpu/fpga to the YARN resource types yarn.io/gpu and yarn.io/fpga. YARN also supports custom resource types and in Hadoop 3.3.1 made it easier for users to plugin in custom resource types. This means users may create a custom resource type that represents a GPU or FPGAs because they want additional logic that YARN the built in versions don't have. Ideally Spark users still just use the generic "gpu" or "fpga" types in Spark. So we should add the ability to change the Spark internal mappings.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org