You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Max Xie (Jira)" <ji...@apache.org> on 2019/12/09 14:07:00 UTC

[jira] [Created] (SPARK-30191) AM should update pending resource request faster when driver lost executor

Max  Xie created SPARK-30191:
--------------------------------

             Summary: AM should update pending resource request faster when driver lost executor
                 Key: SPARK-30191
                 URL: https://issues.apache.org/jira/browse/SPARK-30191
             Project: Spark
          Issue Type: Improvement
          Components: YARN
    Affects Versions: 2.4.4
            Reporter: Max  Xie


I run spark on yarn.  I found that when driver lost its executors because of machine hardware problem and all of service includes nodemanager, executor on the node has killed,  it means that Resourcemanager can't update the containers info on the node until Resourcemanager try to remove the node,   but it always takes 10 mins or longger, and in the meantime, AM don't add the new resource request and driver missing the executors.

So maybe AM should add the factor `numExecutorsExiting` in YarnAllocator's method `

updateResourceRequests`  to optimize it.

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org