You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Thomas Graves (Jira)" <ji...@apache.org> on 2020/02/06 15:49:00 UTC
[jira] [Updated] (SPARK-30749) stage level scheduling: Better
cleanup of Resource profiles
[ https://issues.apache.org/jira/browse/SPARK-30749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated SPARK-30749:
----------------------------------
Summary: stage level scheduling: Better cleanup of Resource profiles (was: Better cleanup of Resource profiles)
> stage level scheduling: Better cleanup of Resource profiles
> -----------------------------------------------------------
>
> Key: SPARK-30749
> URL: https://issues.apache.org/jira/browse/SPARK-30749
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.1.0
> Reporter: Thomas Graves
> Priority: Major
>
> In the initial stage level scheduling jiras we aren't doing a great job of cleaning up datastructures when the ResourceProfile is done being used. This is mostly because its hard to tell when its done being used.
> We should find a way to clean up better.
> Also in the dynamic allocation manager, if the resource profile is not being used anymore we should be more active about getting rid of the executors that are up. Especially if there is a minimum number that is set, we can kill those off.
>
> I think this can be done as followup to the main feature
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org