[jira] [Updated] (SPARK-49905) Use different ShuffleOrigin for the shuffle required from stateful operators

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-49905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-49905:
-----------------------------------
    Labels: pull-request-available  (was: )

> Use different ShuffleOrigin for the shuffle required from stateful operators
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-49905
>                 URL: https://issues.apache.org/jira/browse/SPARK-49905
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL, Structured Streaming
>    Affects Versions: 4.0.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>              Labels: pull-request-available
>
> ShuffleOrigin was introduced to determine whether the shuffle can be adjusted or not, and if it can be, in which way.
>  
> ENSURE_REQUIREMENTS used by the rule `EnsureRequirements` denotes that the shuffle can be adjusted as long as the requirements are still ensured. Here the requirements do not include the number of partitions so AQE can possibly optimize the number (assuming that the number of partitions isn't a strong requirement), but for stateful operators, the number of partitions is a strong requirement and it must not be adjusted.
>  
> To prevent any further modification of AQE to break stateful operator (as same as the reason we introduced StatefulOpClusteredDistribution), we want to explicitly mark the shuffle backing up stateful operator as different ShuffleOrigin.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org