You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Thomas Graves (Jira)" <ji...@apache.org> on 2020/02/06 15:49:00 UTC

[jira] [Created] (SPARK-30749) Better cleanup of Resource profiles

Thomas Graves created SPARK-30749:
-------------------------------------

             Summary: Better cleanup of Resource profiles
                 Key: SPARK-30749
                 URL: https://issues.apache.org/jira/browse/SPARK-30749
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 3.1.0
            Reporter: Thomas Graves


In the initial stage level scheduling jiras we aren't doing a great job of cleaning up datastructures when the ResourceProfile is done being used. This is mostly because its hard to tell when its done being used.

We should find a way to clean up better.

Also in the dynamic allocation manager, if the resource profile is not being used anymore we should be more active about getting rid of the executors that are up. Especially if there is a minimum number that is set, we can kill those off.

 

I think this can be done as followup to the main feature



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org