You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2025/10/17 06:51:00 UTC

[jira] [Created] (SPARK-53941) Support AQE in stateless streaming workloads

Jungtaek Lim created SPARK-53941:
------------------------------------

             Summary: Support AQE in stateless streaming workloads
                 Key: SPARK-53941
                 URL: https://issues.apache.org/jira/browse/SPARK-53941
             Project: Spark
          Issue Type: New Feature
          Components: Structured Streaming
    Affects Versions: 4.1.0
            Reporter: Jungtaek Lim


We have been disabling AQE for streaming workloads since stateful operators do not work with AQE. But applying AQE in stateless workloads is still beneficial and we have been blocking the case too aggressively.

We have been observed streaming workloads which can enjoy the benefits of AQE e.g. stream-static join with large static table, MERGE INTO in ForeachBatch sink, etc. To accelerate the workloads, we would want to support AQE in stateless streaming workloads.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org