[GitHub] liupc opened a new pull request #23842: [SPARK-26927]Fix race condition may cause dynamic allocation not working

[GitBox <gi...@apache.org>]

liupc opened a new pull request #23842: [SPARK-26927]Fix race condition may cause dynamic allocation not working
URL: https://github.com/apache/spark/pull/23842
 
 
   ## What changes were proposed in this pull request?
   
   There is a race condition in the ExecutorAllocationManager that the `SparkListenerExecutorRemoved` event is posted before the `SparkListenerTaskStart` event, which will cause the incorrect result of `executorIds`. Then, when some executor idles, the real executors will be removed even actual executor number is equal to `minNumExecutors` due to the incorrect computation of `newExecutorTotal`(may greater than the `minNumExecutors`), thus may finally causing zero available executors but a wrong number of executorIds was kept in memory.
   
   What's more, even the `SparkListenerTaskEnd` event can not make the fake `executorIds` released, because later idle event for the fake executors can not cause the real removal of these executors, as they are already removed and they are not exist in the `executorDataMap`  of `CoaseGrainedSchedulerBackend`.
   
   For details see https://issues.apache.org/jira/browse/SPARK-26927
   
   This PR is to fix this problem.
   
   ## How was this patch tested?
   
   existUT and added UT
   
   Please review http://spark.apache.org/contributing.html before opening a pull request.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org