[GitHub] SandishKumarHN opened a new pull request #23401: Trigger GC on executor node idle

[GitBox <gi...@apache.org>]

SandishKumarHN opened a new pull request #23401: Trigger GC on executor node idle
URL: https://github.com/apache/spark/pull/23401
 
 
   ## Trigger GC on executor node idle 
   
   Correct me if I'm wrong.
   ### Stage:
         On a large cluster, each stage would have some executors. were a few executors would finish a couple of tasks first and wait for whole stage or remaining tasks to finish which are executed by different executors nodes in a cluster. a stage will only be completed when all tasks in a current stage finish its execution. and the next stage execution has to wait till all tasks of the current stage are completed. 
   
   why don't we trigger GC, when the executor node is waiting for remaining tasks to finish, or when executor node is Idle?
   
   I have proposed a small code snippet which triggers GC when running-tasks are empty and heap usage in current executor node is more than the given threshold.
   
   This could improve performance for long-running spark job's. 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org