[GitHub] [spark] aminebag commented on pull request #41423: [SPARK-43523][CORE] Fix Spark UI LiveTask memory leak

["aminebag (via GitHub)" <gi...@apache.org>]

aminebag commented on PR #41423:
URL: https://github.com/apache/spark/pull/41423#issuecomment-1575506059

   @srowen 
   I have applied and tested the modification you have suggested (replace toList with toArray) and it had no observable impact on the issue. I can still include it in the change If you want.
   
   @sarutak 
   Increasing the value of `spark.scheduler.listenerbus.eventqueue.capacity` would reduce the severity of the leak but I don't think that it would entirely prevent it from happening, especially at moments of peaks of activity. I suppose we would have less event drops by increasing the size of the queue, but tasks would still leak and keep adding-up in memory until eventually there's no more space for the driver to operate.
   What I have noticed with the heap dumps in both the cases of our application in production and the tests, is that the most significant leak is the liveTasks map of the ApplicationStatusListener object. As I have illustrated using the tests, the leak was contained by limiting the size of the liveTasks map. I think that this fix addresses the problem directly, since no matter how many events are dropped, the number of leaked tasks will never exceed the value of `spark.ui.refreshedTasks`. Thus, the application can live for days and we would never worry about liveTasks map taking over all memory. Especially that the role of AppStatusListener is just to provide monitoring information about the state of the application, and when you start to have event drops, AppStatusListener's state is already incoherent.
   This new property `spark.ui.refreshedTasks` is optional and disabled by default and we would only need to enable it when we observe that the AppStatusListener is leaking.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org