[jira] [Created] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

["Matt Cheah (JIRA)" <ji...@apache.org>]

Matt Cheah created SPARK-24248:
----------------------------------

             Summary: [K8S] Use the Kubernetes cluster as the backing store for the state of pods
                 Key: SPARK-24248
                 URL: https://issues.apache.org/jira/browse/SPARK-24248
             Project: Spark
          Issue Type: Improvement
          Components: Kubernetes
    Affects Versions: 2.3.0
            Reporter: Matt Cheah


We have a number of places in KubernetesClusterSchedulerBackend right now that maintains the state of pods in memory. However, the Kubernetes API can always give us the most up to date and correct view of what our executors are doing. We should consider moving away from in-memory state as much as can in favor of using the Kubernetes cluster as the source of truth for pod status. Maintaining less state in memory makes it so that there's a lower chance that we accidentally miss updating one of these data structures and breaking the lifecycle of executors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org