You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Matt Cheah (JIRA)" <ji...@apache.org> on 2018/06/14 23:17:00 UTC

[jira] [Resolved] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

     [ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Cheah resolved SPARK-24248.
--------------------------------
       Resolution: Fixed
    Fix Version/s: 2.4.0

> [K8S] Use the Kubernetes cluster as the backing store for the state of pods
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-24248
>                 URL: https://issues.apache.org/jira/browse/SPARK-24248
>             Project: Spark
>          Issue Type: Improvement
>          Components: Kubernetes
>    Affects Versions: 2.3.0
>            Reporter: Matt Cheah
>            Priority: Major
>             Fix For: 2.4.0
>
>
> We have a number of places in KubernetesClusterSchedulerBackend right now that maintains the state of pods in memory. However, the Kubernetes API can always give us the most up to date and correct view of what our executors are doing. We should consider moving away from in-memory state as much as can in favor of using the Kubernetes cluster as the source of truth for pod status. Maintaining less state in memory makes it so that there's a lower chance that we accidentally miss updating one of these data structures and breaking the lifecycle of executors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org