[jira] [Created] (SPARK-46062) CTE reference node does not inherit the flag `isStreaming` from CTE definition node

["Jungtaek Lim (Jira)" <ji...@apache.org>]

Jungtaek Lim created SPARK-46062:
------------------------------------

             Summary: CTE reference node does not inherit the flag `isStreaming` from CTE definition node
                 Key: SPARK-46062
                 URL: https://issues.apache.org/jira/browse/SPARK-46062
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.5.0, 3.4.1, 3.3.2, 4.0.0
            Reporter: Jungtaek Lim


Looks like this is a long standing bug.

We figured out that CTE reference node would never set the isStreaming flag to true, regardless of the value for flag in CTE definition. The node cannot determine the right value of isStreaming flag by itself (likewise it cannot determine about resolution by itself) but it has no parameter in constructor, hence always takes the default (no children, so batch one).

This may impact some rules which behaves differently depending on isStreaming flag. It would no longer be a problem once CTE reference is replaced with CTE definition at some point in "optimization phase", but all rules in analyzer and optimizer being triggered before the rule takes effect may be impacted.

We probably couldn't sync the flag in real time, but we should sync the flag when we mark CTE reference to be "resolved". The rule `ResolveWithCTE` will be a good place to do that.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org