You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Weiwei Yang (Jira)" <ji...@apache.org> on 2021/03/10 01:10:00 UTC
[jira] [Commented] (YUNIKORN-558) Integrate YuniKorn with Spark K8S
Operator
[ https://issues.apache.org/jira/browse/YUNIKORN-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298466#comment-17298466 ]
Weiwei Yang commented on YUNIKORN-558:
--------------------------------------
Thanks [~phoenixjiangnan] for creating this issue. YK can work with spark-operator today seamlessly. As long as the admission controller is enabled (that adds the schedulerName upon admission), YK automatically recognizes the Spark jobs and schedules them as needed. Deeper integration with Spark operator means a better app lifecycle mgmt, and the support of gang scheduling. A rough thought about the work to be done including:
# [dev] Manage SparkApplication/ScheduledSparkApplication end-to-end lifecycle with app-mgmt plugin
# [dev] Support YK gang scheduling by properly setting up taskGroups info in the Spark app CRD
# [test] e2e test with Spark operator w/ or w/o gang scheduling
# [doc] User instructions
cc folks who might be interested too: [~leftnoteasy], [~sunil.govind@gmail.com], [~wilfreds], [~kmarton], [~ayubpathan], [~rozhang], [~Zhou Kang], [~Huang Ting Yao].
> Integrate YuniKorn with Spark K8S Operator
> ------------------------------------------
>
> Key: YUNIKORN-558
> URL: https://issues.apache.org/jira/browse/YUNIKORN-558
> Project: Apache YuniKorn
> Issue Type: New Feature
> Components: core - scheduler
> Reporter: Bowen Li
> Priority: Major
>
> Integrate YuniKorn with Spark K8S Operator [https://github.com/GoogleCloudPlatform/spark-on-k8s-operator]
>
> Using Spark K8S Operator is a standard way to run Spark applications on K8S. The native integration will make using Yunikorn more easier to use and get more exposure to OSS community
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org