[jira] [Created] (SPARK-49070) TransformWithStateExec.initialState is rewritten incorrectly to produce invalid query plan

["L. C. Hsieh (Jira)" <ji...@apache.org>]

L. C. Hsieh created SPARK-49070:
-----------------------------------

             Summary: TransformWithStateExec.initialState is rewritten incorrectly to produce invalid query plan
                 Key: SPARK-49070
                 URL: https://issues.apache.org/jira/browse/SPARK-49070
             Project: Spark
          Issue Type: Bug
          Components: SQL, Structured Streaming
    Affects Versions: 4.0.0
            Reporter: L. C. Hsieh


SPARK-47363 added the support for users to provide initial state for streaming query. Such query operators like `TransformWithStateExec` might have `hasInitialState` as false which means the initial state related parameters are not used. But when query planner applies rules on the query, it will still apply on the initial state query plan. When `hasInitialState` is false, some related parameters  like `initialStateGroupingAttrs` are invalid and some rules will use these invalid parameters to transform the initial state query plan.

For example, `EnsureRequirements` may apply invalid Sort and Exchange on the initial query plan. We encountered these invalid query plan in our extension rules. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org