[GitHub] [spark] tgravescs commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes

[GitBox <gi...@apache.org>]

tgravescs commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-814126917


   yeah its unclear to me how this fits in with the rest of the shuffle work going on and to be honest I haven't kept up enough to comment without reviewing that again to see where everything is at.  This is also just a huge PR to review in one shot.  
   
   I think it would be nice to have a remote shuffle service solution for k8s in spark but it should also be pluggable and we should chose what that looks like.  My reasons are:
   
   - we have one for other cluster managers and having one for k8s allows us to make sure it integrates well with everything like dynamic allocation and other features.   The version internally should use the same plugin api others use.  
   - ease of use
   - make sure its tested 
   - ability to more easily change it, enhance it and keep it in sync.  If there are other ones great, but many times those are in private repos and hard to get changes in.
   
   I agree with the point as to why this one should go in and I think we should discuss and see what makes the most sense.  In some ways you also have to look at who is going to put in the effort to get it merged and also maintain it.  Shuffle work has been going on for a long time now and it seems like we get a few prs and then stops for a while. Obviously people have other things going on too and its a lot of effort.
   
   In my mind it comes down is it beneficial to the majority of users, who is going to put in the effort and is it going to be maintained if its pulled in.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org