[jira] [Updated] (SPARK-55555) Support heterogeneous K8s executor management

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-55555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun updated SPARK-55555:
----------------------------------
    Labels: releasenotes  (was: )

> Support heterogeneous K8s executor management
> ---------------------------------------------
>
>                 Key: SPARK-55555
>                 URL: https://issues.apache.org/jira/browse/SPARK-55555
>             Project: Spark
>          Issue Type: Umbrella
>          Components: Kubernetes
>    Affects Versions: 4.2.0
>            Reporter: Dongjoon Hyun
>            Assignee: Dongjoon Hyun
>            Priority: Critical
>              Labels: releasenotes
>             Fix For: 4.2.0
>
>
> Historically, the Apache Spark manages executors in the well-known ways via (1) Dynamic Allocation, (2) Resource profiles, (3) Executor Rolling. However, many code patch still assumes homogeneous executors.
> This issue aims to cover more advanced resource managements by taking advantage of new K8s features. For example,
> - We can resize CPU and Memory requests and limits of executors via *In-Place Pod Update* dynamically and systematically in advance.
> - We can scale-up the PVC volumes via *AllowVolumeExpansion*.
> - We can introduce a new kind of executor type, *Recovery-mode Executor*, to allow only a single task per executor JVM in case of OOM.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org