(spark) branch branch-3.3 updated: [SPARK-46006][YARN][FOLLOWUP] YarnAllocator set target executor number to 0 to cancel pending allocate request when driver stop

[ya...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

yao pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
     new b53090436590 [SPARK-46006][YARN][FOLLOWUP] YarnAllocator set target executor number to 0 to cancel pending allocate request when driver stop
b53090436590 is described below

commit b53090436590edf5831cce2cc39dc2f994246e45
Author: Angerszhuuuu <an...@gmail.com>
AuthorDate: Tue Nov 28 11:04:14 2023 +0800

    [SPARK-46006][YARN][FOLLOWUP] YarnAllocator set target executor number to 0 to cancel pending allocate request when driver stop
    
    ### What changes were proposed in this pull request?
    YarnAllocator set target executor number to 0 to cancel pending allocate request when driver stop
    Now for this issue we do:
    
    1. AllocationFailure should not be treated as exitCausedByApp when driver is shutting down https://github.com/apache/spark/pull/38622
    2. Avoid new allocation requests when sc.stop stuck https://github.com/apache/spark/pull/43906
    3. Cancel pending allocation request, this pr https://github.com/apache/spark/pull/44036
    
    ### Why are the changes needed?
    Avoid unnecessary allocate request
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    MT
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No
    
    Closes #44036 from AngersZhuuuu/SPARK-46006-FOLLOWUP.
    
    Authored-by: Angerszhuuuu <an...@gmail.com>
    Signed-off-by: Kent Yao <ya...@apache.org>
    (cherry picked from commit dbc8756bdac823be42ed10bc011415f405905497)
    Signed-off-by: Kent Yao <ya...@apache.org>
---
 .../src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala  | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
index 58cf8765bf1f..4b028f0e5f7f 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
@@ -355,7 +355,10 @@ private[yarn] class YarnAllocator(
     this.hostToLocalTaskCountPerResourceProfileId = hostToLocalTaskCountPerResourceProfileId
 
     if (resourceProfileToTotalExecs.isEmpty) {
-      targetNumExecutorsPerResourceProfileId.clear()
+      // Set target executor number to 0 to cancel pending allocate request.
+      targetNumExecutorsPerResourceProfileId.keys.foreach { rp =>
+        targetNumExecutorsPerResourceProfileId(rp) = 0
+      }
       allocatorNodeHealthTracker.setSchedulerExcludedNodes(excludedNodes)
       true
     } else {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org