[GitHub] [spark] mccheah commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files

[GitBox <gi...@apache.org>]

mccheah commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files
URL: https://github.com/apache/spark/pull/24892#issuecomment-518448635
 
 
   I may be misunderstanding the situation, but I don't see why we can't use the task attempt id to deduplicate multiple attempts of running the same map task. Why do we need to introduce a second identifier when we already have an identifier for representing a different attempt of the same task? I'd imagine re-submitting the indeterministic map stage would result in fresh task attempt identifiers. Then on the read side, we'd just have to know which attempt we have to use on the reduce side.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org