You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/10/22 22:33:00 UTC
[jira] [Updated] (SPARK-60480) Avoid scanning healthy executor pods in ExecutorPodsLifecycleManager
[ https://issues.apache.org/jira/browse/SPARK-60480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-60480:
-----------------------------------
Labels: pull-request-available (was: )
> Avoid scanning healthy executor pods in ExecutorPodsLifecycleManager
> --------------------------------------------------------------------
>
> Key: SPARK-60480
> URL: https://issues.apache.org/jira/browse/SPARK-60480
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 5.0.0
> Reporter: Victor Sunderland
> Priority: Major
> Labels: pull-request-available
>
> {{ExecutorPodsLifecycleManager.onNewSnapshots()}} scans the full executor pod map of every snapshot twice: once to collect failed executors and once to handle final and inactive pods. Healthy (pending or running) pods never need lifecycle handling, so with many executors and frequent watch updates most of this work is wasted.
> Each {{ExecutorPodsSnapshot}} can keep an index of its final and inactive pods, maintained incrementally in {{withUpdate()}}, so the manager only visits the pods it acts on. Every snapshot is still processed, so failures seen in intermediate snapshots and retries of failed pod deletions are preserved. Missing-pod detection keeps using the full pod map.
> In a local synthetic benchmark with 10,000 executor pods and 100 watch updates per batch, snapshot updates plus lifecycle processing for healthy-pod batches dropped from 14.9 ms to 0.011 ms.
> This is similar to SPARK-55963, which optimized snapshot traversal in {{ExecutorPodsAllocator}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org