[jira] [Resolved] (SPARK-53941) Support AQE in stateless streaming workloads

["Jungtaek Lim (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-53941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim resolved SPARK-53941.
----------------------------------
    Fix Version/s: 4.1.0
       Resolution: Fixed

Issue resolved by pull request 52642
[https://github.com/apache/spark/pull/52642]

> Support AQE in stateless streaming workloads
> --------------------------------------------
>
>                 Key: SPARK-53941
>                 URL: https://issues.apache.org/jira/browse/SPARK-53941
>             Project: Spark
>          Issue Type: New Feature
>          Components: Structured Streaming
>    Affects Versions: 4.1.0
>            Reporter: Jungtaek Lim
>            Assignee: Jungtaek Lim
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.1.0
>
>
> We have been disabling AQE for streaming workloads since stateful operators do not work with AQE. But applying AQE in stateless workloads is still beneficial and we have been blocking the case too aggressively.
> We have been observed streaming workloads which can enjoy the benefits of AQE e.g. stream-static join with large static table, MERGE INTO in ForeachBatch sink, etc. To accelerate the workloads, we would want to support AQE in stateless streaming workloads.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org