You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Nandini (Jira)" <ji...@apache.org> on 2021/10/14 13:18:00 UTC
[jira] [Issue Comment Deleted] (SPARK-37007)
ExecutorAllocationManager schedule() does not use
spark.dynamicAllocation.executorIdleTimeout
[ https://issues.apache.org/jira/browse/SPARK-37007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nandini updated SPARK-37007:
----------------------------
Comment: was deleted
(was: Hello Team,
I would like to work on this jira.)
> ExecutorAllocationManager schedule() does not use spark.dynamicAllocation.executorIdleTimeout
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-37007
> URL: https://issues.apache.org/jira/browse/SPARK-37007
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.1, 3.2.0
> Reporter: Nandini
> Priority: Minor
>
> The ExecutorAllocationManager removes idle executors after the configured spark.dynamicAllocation.executorIdleTimeout but in the schedule() it does not use the same configuration.
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L249
> The value for intervalMillis is 100 and timeunit is ms. Hence you see this log approximately 10 times in a second.
> | executor.scheduleWithFixedDelay(scheduleTask, 0, intervalMillis, TimeUnit.MILLISECONDS)
> In older versions it was at info level
> [https://github.com/apache/spark/blob/v2.4.0/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L454]
> In the latest versions (and master) it has been changed to debug
> [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L540]
> The change request for the above -
> [https://github.com/apache/spark/commit/3584d849438ad48ff54af3c982c124a8443dc590]
> However, this check for executors to be removed should be using spark.dynamicAllocation.executorIdleTimeout instead of https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L153
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org