You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "William Benton (JIRA)" <ji...@apache.org> on 2016/11/07 02:27:58 UTC
[jira] [Comment Edited] (SPARK-18278) Support native submission of
spark jobs to a kubernetes cluster
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15642858#comment-15642858 ]
William Benton edited comment on SPARK-18278 at 11/7/16 2:27 AM:
-----------------------------------------------------------------
[~srowen] Currently {{ExternalClusterManager}} is Spark-private, so there isn't a great way to implement a new scheduler backend outside of Spark proper. I think it would be great if an extension API for new cluster managers were public and developers could work with it with some expectation of stability! But even if this API were exposed and (relatively) stable, I think there's a good argument that if any cluster managers besides standalone are to live in Spark proper that a Kubernetes scheduler should be there, too.
(Why draw the line just past Mesos and YARN when Kubernetes also enjoys a large community and many deployments? And if the Spark community is to draw the line somewhere, why not draw it around the standalone scheduler? If we're using the messaging connectors in Bahir as an example, then historical precedent isn't an argument for keeping existing schedulers in Spark proper.)
was (Author: willbenton):
[~srowen] Currently {{ExternalClusterManager}} is Spark-private, so there isn't a great way to implement a new scheduler backend outside of Spark proper. I think it would be great if an extension API for new cluster managers were public and developers could work with it with some expectation of stability! But even if this API were exposed and (relatively) stable, I I think there's a good argument that if any cluster managers besides standalone are to live in Spark proper that a Kubernetes scheduler should be there, too.
(Why draw the line just past Mesos and YARN when Kubernetes also enjoys a large community and many deployments? And if the Spark community is to draw the line somewhere, why not draw it around the standalone scheduler? If we're using the messaging connectors in Bahir as an example, then historical precedent isn't an argument for keeping existing schedulers in Spark proper.)
> Support native submission of spark jobs to a kubernetes cluster
> ---------------------------------------------------------------
>
> Key: SPARK-18278
> URL: https://issues.apache.org/jira/browse/SPARK-18278
> Project: Spark
> Issue Type: Umbrella
> Components: Build, Deploy, Documentation, Scheduler, Spark Core
> Affects Versions: 2.2.0
> Reporter: Erik Erlandson
>
> A new Apache Spark sub-project that enables native support for submitting Spark applications to a kubernetes cluster. The submitted application runs in a driver executing on a kubernetes pod, and executors lifecycles are also managed as pods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org