[GitHub] [spark] okimcoder commented on pull request #28656: [SPARK-31837][CORE] Shift to the new highest locality level if there is when recomputeLocality

[GitBox <gi...@apache.org>]

okimcoder commented on pull request #28656:
URL: https://github.com/apache/spark/pull/28656#issuecomment-635650972


   @Ngone51 
   Thanks for looking into this.
   
   Let's make a few corrections to the description:
   1. This happened with previous delay scheduling as well (although likely less noticeable as @tgravescs  pointed out)
   2. It won't "always" use ANY for "all" tasks in the taskset. For example if a single executor is added that is NODE_LOCAL and is used for scheduling, the very first resourceOffer would switch the taskset to NODE_LOCAL.
   
   I do see how this handles the race condition on startup, although a more specific handling of that case may be preferable (@tgravescs also mentioned).
   It seems questionable whether executor added should instantly reset the locality levels.
   
   Deferring to @tgravescs  and @cloud-fan to make the call here.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org