You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Andrew Or (JIRA)" <ji...@apache.org> on 2014/11/15 00:58:34 UTC

[jira] [Closed] (SPARK-4214) With dynamic allocation, avoid outstanding requests for more executors than pending tasks need

     [ https://issues.apache.org/jira/browse/SPARK-4214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Or closed SPARK-4214.
----------------------------
       Resolution: Fixed
    Fix Version/s: 1.2.0

> With dynamic allocation, avoid outstanding requests for more executors than pending tasks need
> ----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-4214
>                 URL: https://issues.apache.org/jira/browse/SPARK-4214
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core, YARN
>    Affects Versions: 1.2.0
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>             Fix For: 1.2.0
>
>
> Dynamic allocation tries to allocate more executors while we have pending tasks remaining.  Our current policy can end up with more outstanding executor requests than needed to fulfill all the pending tasks.  Capping the executor requests to the number of cores needed to fulfill all pending tasks would make dynamic allocation behavior less sensitive to settings for maxExecutors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org