[jira] [Comment Edited] (SPARK-24793) Make spark-submit more useful with k8s

["Stavros Kontopoulos (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665578#comment-16665578 ] 

Stavros Kontopoulos edited comment on SPARK-24793 at 10/26/18 7:15 PM:
-----------------------------------------------------------------------

From a quick glance you can't just use the k8s backend to check status of the driver. Standalone and mesos mode can support this because they are using the rest cient which is a common api always available at spark core. We cant add k8s dependency by default at that point of code. You then either use reflection if k8s master is passed to load a class from the backend side or query the K8s api server by extending that rest client and mapping pod status to drivers status to keep UX the same.


was (Author: skonto):
From a quick glance you can't just use the k8s backend to check status of the driver. Standalone and mesos mode can support this because they are using the rest cient which is a common api always available at spark core. We cant add k8s dependency by default at that point of code. You then either use reflection or hit the api server with a rest api.

> Make spark-submit more useful with k8s
> --------------------------------------
>
>                 Key: SPARK-24793
>                 URL: https://issues.apache.org/jira/browse/SPARK-24793
>             Project: Spark
>          Issue Type: Improvement
>          Components: Kubernetes
>    Affects Versions: 2.3.0
>            Reporter: Anirudh Ramanathan
>            Assignee: Anirudh Ramanathan
>            Priority: Major
>
> Support controlling the lifecycle of Spark Application through spark-submit. 
> For example:
> {{ 
>   --kill app_name           If given, kills the driver specified.
>   --status app_name      If given, requests the status of the driver specified.
> }}
> Potentially also --list to list all spark drivers running.
> Given that our submission client can actually launch jobs into many different namespaces, we'll need an additional specification of the namespace through a --namespace flag potentially.
> I think this is pretty useful to have instead of forcing a user to use kubectl to manage the lifecycle of any k8s Spark Application.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org