[jira] [Created] (SPARK-32922) Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures

["Min Shen (Jira)" <ji...@apache.org>]

Min Shen created SPARK-32922:
--------------------------------

             Summary: Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures
                 Key: SPARK-32922
                 URL: https://issues.apache.org/jira/browse/SPARK-32922
             Project: Spark
          Issue Type: Sub-task
          Components: Shuffle, Spark Core
    Affects Versions: 3.1.0
            Reporter: Min Shen


With the extended MapOutputTracker, the reducers can now get the task input data from the merged shuffle partitions for more efficient shuffle data fetch. The reducers should also be able to fallback to fetching the original unmarked blocks if it encounters failures when fetching the merged shuffle partitions.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org