[GitHub] [spark] Ngone51 commented on pull request #32136: [WIP][SPARK-35022][CORE] Task Scheduling Plugin in Spark

[GitBox <gi...@apache.org>]

Ngone51 commented on pull request #32136:
URL: https://github.com/apache/spark/pull/32136#issuecomment-819332344


   > ... So (a) looks like using locality for state store location and (b) is that locality cannot guarantee actual location. Right? 
   
   Yes.
   
   > 1. it uses previous state store location as locality, so if no previous location info, we still let Spark pick up executor arbitrarily.
   
   So, how would the plugin help when there's no previous location info? 
   
   > 2. It depends on initial chosen executor-state store mapping. So if Spark choose a sub-optimal mapping, locality doesn't work well for later batches. 
   
   I think this is the point that matches my point of case b above.
   
   But looking at the code, it seems the plugin is still applied after locality scheduling? 
   
   > 3. Forcibly assigning state stores to executors can possibly lead to unreasonable scheduling decision. For example, we don't know if the executor satisfy resource requirement.
   
   I don't get this. Do you mean some executors may not be suitable for having a state store?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org