[jira] [Resolved] (SPARK-41198) Streaming query metrics is broken with CTE

["Jungtaek Lim (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-41198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim resolved SPARK-41198.
----------------------------------
    Fix Version/s: 3.4.0
       Resolution: Fixed

Issue resolved by pull request 38717
[https://github.com/apache/spark/pull/38717]

> Streaming query metrics is broken with CTE
> ------------------------------------------
>
>                 Key: SPARK-41198
>                 URL: https://issues.apache.org/jira/browse/SPARK-41198
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 3.2.2, 3.4.0, 3.3.1
>            Reporter: Jungtaek Lim
>            Assignee: Jungtaek Lim
>            Priority: Major
>             Fix For: 3.4.0
>
>
> We have observed a case the metrics are not available for the streaming query which contains CTE.
> Looks like CTE was inlined in analysis phase in Spark 3.1.x and it was changed to be inlined in optimization phase in Spark 3.2.x. ProgressReporter depends on analyzed plan, hence the change made ProgressReporter to see CTE nodes, which ends up with having different number of leaf nodes between analyzed plan and executed plan.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org