You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Thomas Graves (Jira)" <ji...@apache.org> on 2020/05/28 14:27:00 UTC

[jira] [Created] (SPARK-31856) Handle locality wait reset better when executors added

Thomas Graves created SPARK-31856:
-------------------------------------

             Summary: Handle locality wait reset better when executors added
                 Key: SPARK-31856
                 URL: https://issues.apache.org/jira/browse/SPARK-31856
             Project: Spark
          Issue Type: Improvement
          Components: Scheduler
    Affects Versions: 3.1.0
            Reporter: Thomas Graves


We found a corner case with the new locality algorithm where we were not affectively resetting the locality when you started without executors.  See [https://github.com/apache/spark/pull/28656/]

In that fix we reset the locality when a new executor is added and it changes the valid locality levels.  This could reset it when you don't want to. for instance if the task set has been around for quite a while and you have other executors it could put tasks on. Resetting it back to the most local locality level could cause delays in the tasks getting scheduled that have been waiting for more then the locality wait. 

This goes back to the new algorithm tries to use heuristic on this but its not perfect.  Ideally I think we would still track per executor but that is a lot more change.

Investigate to see if we can handle this better. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org