[jira] [Updated] (SPARK-46006) YarnAllocator miss clean targetNumExecutorsPerResourceProfileId after YarnSchedulerBackend call stop

["angerszhu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-46006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

angerszhu updated SPARK-46006:
------------------------------
    Description: 
We meet a case that user call sc.stop() after run all custom code, but stuck in some place. 

Cause below situation
 # User call sc.stop()
 # 
sc.stop()卡住
ApplicationMaster 还没有进入finish， 还会继续调用YarnAllocator.allocateResources()
因为driver端口已经关闭，allocateResource还在继续申请新的executor失败
触发 Max number of executor failures

> YarnAllocator miss clean targetNumExecutorsPerResourceProfileId after YarnSchedulerBackend call stop
> ----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-46006
>                 URL: https://issues.apache.org/jira/browse/SPARK-46006
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 3.1.3, 3.2.4, 3.3.2, 3.4.1, 3.5.0
>            Reporter: angerszhu
>            Priority: Major
>             Fix For: 3.4.2, 4.0.0, 3.5.1
>
>
> We meet a case that user call sc.stop() after run all custom code, but stuck in some place. 
> Cause below situation
>  # User call sc.stop()
>  # 
> sc.stop()卡住
> ApplicationMaster 还没有进入finish， 还会继续调用YarnAllocator.allocateResources()
> 因为driver端口已经关闭，allocateResource还在继续申请新的executor失败
> 触发 Max number of executor failures



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org