[jira] [Updated] (SPARK-35036) Improve push based shuffle to work with AQE by fetching partial map indexes for a reduce partition

["Min Shen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-35036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Min Shen updated SPARK-35036:
-----------------------------
        Parent:     (was: SPARK-30602)
    Issue Type: New Feature  (was: Sub-task)

> Improve push based shuffle to work with AQE by fetching partial map indexes for a reduce partition
> --------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-35036
>                 URL: https://issues.apache.org/jira/browse/SPARK-35036
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 3.1.1
>            Reporter: Venkata krishnan Sowrirajan
>            Priority: Major
>
> Currently when both Push based shuffle and AQE is enabled and when partial set of map indexes are requested to MapOutputTracker this is delegated the regular shuffle instead of push based shuffle reading map blocks. This is because blocks from mapper in push based shuffle are merged out of order due to which its hard to only get the matching blocks of the reduce partition for the requested start and end map indexes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org