[jira] [Created] (SPARK-60480) Avoid scanning healthy executor pods in ExecutorPodsLifecycleManager

["Victor Sunderland (Jira)" <ji...@apache.org>]

Victor Sunderland created SPARK-60480:
-----------------------------------------

             Summary: Avoid scanning healthy executor pods in ExecutorPodsLifecycleManager
                 Key: SPARK-60480
                 URL: https://issues.apache.org/jira/browse/SPARK-60480
             Project: Spark
          Issue Type: Improvement
          Components: Kubernetes
    Affects Versions: 5.0.0
            Reporter: Victor Sunderland


{{ExecutorPodsLifecycleManager.onNewSnapshots()}} scans the full executor pod map of every snapshot twice: once to collect failed executors and once to handle final and inactive pods. Healthy (pending or running) pods never need lifecycle handling, so with many executors and frequent watch updates most of this work is wasted.

Each {{ExecutorPodsSnapshot}} can keep an index of its final and inactive pods, maintained incrementally in {{withUpdate()}}, so the manager only visits the pods it acts on. Every snapshot is still processed, so failures seen in intermediate snapshots and retries of failed pod deletions are preserved. Missing-pod detection keeps using the full pod map.

In a local synthetic benchmark with 10,000 executor pods and 100 watch updates per batch, snapshot updates plus lifecycle processing for healthy-pod batches dropped from 14.9 ms to 0.011 ms.

This is similar to SPARK-55963, which optimized snapshot traversal in {{ExecutorPodsAllocator}}.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org