You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Thomas Graves (JIRA)" <ji...@apache.org> on 2019/05/16 16:22:00 UTC
[jira] [Comment Edited] (SPARK-27373) Design: Kubernetes support
for GPU-aware scheduling
[ https://issues.apache.org/jira/browse/SPARK-27373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16841506#comment-16841506 ]
Thomas Graves edited comment on SPARK-27373 at 5/16/19 4:21 PM:
----------------------------------------------------------------
for the kubernetes side, it has 2 options for requesting containers: 1) pod templates, 2) through normal spark and spark.kubernetes configs
For adding in the spark resource support, we can take the spark configs spark.\{driver/executor}.resource.\{resourceName}.count and combine this with a new config for the vendor name like spark.\{driver/executor}.resource.\{resourceName}.vendor to match the device plugin support from k8s ( [https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)] and add that to the PodBuilder.
We could make the vendor config kubernetes specific, but I'm thinking we leave it generic and then just state its only supported on kubernetes right now. Depending on the setup, I could see this being useful for say YARN since yarn support attributes and vendor could be an attribute
spark already has functionality to override and add certain things in the pod templates so we can use similar functionality with the resources. So we can support both the pod templates and the configs the same way.
was (Author: tgraves):
for the kubernetes side, it has 2 options for requesting containers: 1) pod templates, 2) through normal spark and spark.kubernetes configs
For adding in the spark resource support, we can take the spark configs spark.\{driver/executor}.resource.\{resourceName}.count and combine this with a new config for the vendor name like spark.\{driver/executor}.resource.\{resourceName}.vendor to match the device plugin support from k8s ( [https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)] and add that to the PodBuilder.
spark already has functionality to override and add certain things in the pod templates so we can use similar functionality with the resources. So we can support both the pod templates and the configs the same way.
> Design: Kubernetes support for GPU-aware scheduling
> ---------------------------------------------------
>
> Key: SPARK-27373
> URL: https://issues.apache.org/jira/browse/SPARK-27373
> Project: Spark
> Issue Type: Sub-task
> Components: Kubernetes
> Affects Versions: 3.0.0
> Reporter: Xiangrui Meng
> Assignee: Thomas Graves
> Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org