[jira] [Commented] (SPARK-27373) Design: Kubernetes support for GPU-aware scheduling

["Thomas Graves (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-27373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16841506#comment-16841506 ] 

Thomas Graves commented on SPARK-27373:
---------------------------------------

for the kubernetes side, it has 2 options for requesting containers: 1) pod templates, 2) through normal spark and spark.kubernetes configs

For adding in the spark resource support, we can take the spark configs spark.\{driver/executor}.resource.\{resourceName}.count and combine this with a new config for the vendor name like spark.\{driver/executor}.resource.\{resourceName}.vendor to match the device plugin support from k8s ( [https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)] and add that to the PodBuilder.

spark already has functionality to override and add certain things in the pod templates so we can use similar functionality with the resources. So we can support both the pod templates and the configs the same way.

> Design: Kubernetes support for GPU-aware scheduling
> ---------------------------------------------------
>
>                 Key: SPARK-27373
>                 URL: https://issues.apache.org/jira/browse/SPARK-27373
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Kubernetes
>    Affects Versions: 3.0.0
>            Reporter: Xiangrui Meng
>            Assignee: Thomas Graves
>            Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org