You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Thomas Graves (Jira)" <ji...@apache.org> on 2020/10/29 14:17:00 UTC
[jira] [Commented] (SPARK-33288) Support k8s cluster manager with
stage level scheduling
[ https://issues.apache.org/jira/browse/SPARK-33288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17222917#comment-17222917 ]
Thomas Graves commented on SPARK-33288:
---------------------------------------
note there will be a couple of caveats here in that if you have shuffle data on all the nodes from one stage and you go to change the resource profile for another stage if your cluster doesn't have enough nodes that could end up hanging. With normal yarn dynamic allocation with external shuffle, the other executors will go away since no shuffle data on them so you don't have this issue as much.
> Support k8s cluster manager with stage level scheduling
> -------------------------------------------------------
>
> Key: SPARK-33288
> URL: https://issues.apache.org/jira/browse/SPARK-33288
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
> Affects Versions: 3.1.0
> Reporter: Thomas Graves
> Priority: Major
>
> Kubernetes supports dynamic allocation via the {{spark.dynamicAllocation.shuffleTracking.enabled}}
> {{config, we can add support for stage level scheduling when that is turned on. }}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org