[GitHub] [spark] mridulm edited a comment on pull request #34461: [SPARK-37023][CORE] Avoid fetching merge status when shuffleMergeEnabled is false for a shuffleDependency during retry

[GitBox <gi...@apache.org>]

mridulm edited a comment on pull request #34461:
URL: https://github.com/apache/spark/pull/34461#issuecomment-964557253


   @Ngone51 For the example you gave, namely:
   stage1/attempt1 (parent) -> stage2/attempt2 (child)  -- fetch failed --> stage1/attempt2 -> stage2/attempt2
   
   Here, we have two cases:
   
   * If stage1 was non determinate stage - we throw away all merged output from stage1/attempt1 when we run stage1/attempt2.
     * This happens via `shuffleDep.newShuffleMergeState()` which is invoked from beginning of [submitMissingTasks](https://github.com/apache/spark/blob/c4e975e175c01f67ece7ae492a79554ad1b44106/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1374)
   * If stage1 was deterministic stage - we disable shuffle push for stage1/attempt2, and run stage1/attempt2.
     * This happens in `submitMissingTasks` if it is already merged finalized, see [here](https://github.com/apache/spark/blob/c4e975e175c01f67ece7ae492a79554ad1b44106/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1403).
   
   In all of these combinations, whether stage2/attemptN uses merged output or not must only be based on whether merged output exists for it to use - based on parent stage.
   Currently, we are using a single variable to control behavior at both mapper side (push side) and reducer side (using merged output) - this was fine initially - but given for deterministic parent stages and retried child stages, we should make them separate.
   
   IMO we should add a new variable, say `useMergedShuffleInput` - which will be based on whether there exists merged output from parent or not for that shuffle dep at time of launch of child stage.
   
   Thoughts @Victsm, @zhouyejoe, @otterc, @rmcyang, @venkata91 ?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org