[GitHub] rdblue commented on a change in pull request #23401: [SPARK-26513][Core] : Trigger GC on executor node idle

[GitBox <gi...@apache.org>]

rdblue commented on a change in pull request #23401: [SPARK-26513][Core] : Trigger GC on executor node idle
URL: https://github.com/apache/spark/pull/23401#discussion_r245070406
 
 

 ##########
 File path: core/src/main/scala/org/apache/spark/executor/Executor.scala
 ##########
 @@ -628,6 +630,22 @@ private[spark] class Executor(
           }
       } finally {
         runningTasks.remove(taskId)
+        if(idleGCEnabled) {
+          if (runningTasks.isEmpty) {
 
 Review comment:
   I think that the concern is that the executor doesn't know when it will receive another task. Consider the case where an executor has 1 core and high memory usage due to broadcast tables. This would trigger a GC between each task, which could delay the entire job.
   
   I think it would be better to coordinate with the driver, which knows whether there are more tasks in a stage (or if there are multiple concurrent stages). Seems to me that a stage boundary is the right time to optimistically trigger GC. Of course, that only works if the executor is going to keep running.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org