You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Stijn De Haes (Jira)" <ji...@apache.org> on 2020/10/08 13:53:00 UTC
[jira] [Commented] (SPARK-27696) kubernetes driver pod not deleted
after finish.
[ https://issues.apache.org/jira/browse/SPARK-27696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210232#comment-17210232 ]
Stijn De Haes commented on SPARK-27696:
---------------------------------------
I think it is best to have TTL controller for finished resources in kubernetes also work for pods.
[https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/]
Atm this is an alpha level feature that only works for jobs, maybe we can extend it to pods however
> kubernetes driver pod not deleted after finish.
> -----------------------------------------------
>
> Key: SPARK-27696
> URL: https://issues.apache.org/jira/browse/SPARK-27696
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes, Spark Core
> Affects Versions: 2.4.0
> Reporter: Henry Yu
> Priority: Minor
>
> When submit to k8s, driver pod not deleted after job completion.
> While k8s checks driver pod name not existing, It is especially painful when we use workflow tool to resubmit the failed spark job. (by the way, client always exist with 0 is another painful issue)
> I have fix this with a new config spark.kubernetes.submission.deleteCompletedPod=true in our home maintained spark version.
> Do you guys have more insights or I can make a pr on this issue?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org