[GitHub] Ngone51 commented on issue #23377: [SPARK-26439][CORE][WIP] Introduce WorkerOffer reservation mechanism for Barrier TaskSet

[GitBox <gi...@apache.org>]

Ngone51 commented on issue #23377: [SPARK-26439][CORE][WIP] Introduce WorkerOffer reservation mechanism for Barrier TaskSet
URL: https://github.com/apache/spark/pull/23377#issuecomment-451041130
 
 
   > such a big change to the scheduler
   
   It may be a big change for Barrier TaskSet, but remains no difference for non-barrier TaskSet yet.
   
    > by itself the improvement here is just for having multiple concurrent barrier stages, still with static allocation?
   
   Yes, and I think the improvement here could also handle a mix sitution of non-barrier & barrier TaskSets.
   Dynamic allocation would be the next step if this change could be accepted.
   
   > Have you looked at gang scheduling in yarn?
   
   Sorry, no. But I know YARN has reservation mechanism for resource, too. I'll have a look for YARN gang scheduling. 
   
   > I think at this stage, we might benefit more from a high-level design discussion than from code?
   
   Agreed. And it would be better if we could get more people in. Let me try @jiangxb1987 @jerryshao 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org