[GitHub] [spark] cloud-fan edited a comment on issue #24110: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files

[GitBox <gi...@apache.org>]

cloud-fan edited a comment on issue #24110: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
URL: https://github.com/apache/spark/pull/24110#issuecomment-475780104
 
 
   > All the block id should be extended by adding attempt id, not only the indeterminate ones.
   
   That's what I expect. My initial idea is to extend the shuffle id with a generation id, and shuffle id becomes a (int, int) instead of a single int. With the shuffle id change, the shuffle block id also changes, as well as other places that refer to shuffle id, e.g. map output tracker, the shuffle file name maybe.
   
   I think it's a much simpler design comparing to doing different things for different deterministic levels.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org