[GitHub] [spark] dongjoon-hyun commented on pull request #34864: [SHUFFLE] [WIP] Prototype: store shuffle file on external storage like S3

[GitBox <gi...@apache.org>]

dongjoon-hyun commented on pull request #34864:
URL: https://github.com/apache/spark/pull/34864#issuecomment-992740381


   You are completely wrong because you already know the worker decommission feature.
   >  but it will not work well when there is shuffle data distributed on many executors (those executors cannot be released).
   
   You should mention this in the PR description explicitly instead of misleading the users.
   > The work here (storing shuffle data on S3) does not conflict with worker decommission feature. The eventual goal is to store shuffle data on S3 or other external storage directly. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org