[jira] [Updated] (SPARK-60525) Wrap RelationChanges directly in NamedStreamingRelation for streaming CHANGES queries

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-60525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-60525:
-----------------------------------
    Labels: pull-request-available  (was: )

> Wrap RelationChanges directly in NamedStreamingRelation for streaming CHANGES queries
> -------------------------------------------------------------------------------------
>
>                 Key: SPARK-60525
>                 URL: https://issues.apache.org/jira/browse/SPARK-60525
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.2.0
>            Reporter: Anish Mahto
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.4.0
>
>
> When `spark.sql.streaming.queryEvolution.enableSourceEvolution=true`, every streaming source must be named, which SQL users do with `IDENTIFIED BY`. For streaming `CHANGES` queries, combining `IDENTIFIED BY` with an alias or watermark fails analysis:
> ```sql
> SELECT * FROM STREAM cat.t CHANGES FROM VERSION 1 IDENTIFIED BY src AS c;
> SELECT * FROM STREAM cat.t CHANGES FROM VERSION 1 IDENTIFIED BY src
>   WATERMARK _commit_timestamp DELAY OF INTERVAL 1 MINUTE;
> ```
> For a `CHANGES` query, `AstBuilder.visitStreamTableName` builds the `NamedStreamingRelation` over the aliased and watermarked plan, not over `RelationChanges`. With both clauses, the plan looks roughly like:
> ```
> NamedStreamingRelation(src)
> +- EventTimeWatermark
>    +- SubqueryAlias(c)
>       +- RelationChanges
> ```
> `RelationChanges` resolves to `SubqueryAlias(cat.t, StreamingRelationV2)`. The `NameStreamingSources` analysis rule only removes `NamedStreamingRelation` and passes the name down when its child is a streaming relation, or a single `SubqueryAlias` directly over one. Here the watermark, the alias, or both sit in between, so the rule never matches. The `NamedStreamingRelation` is left in the plan unresolved, and analysis fails even though the statement is valid. For `SELECT *`, the error is `INVALID_USAGE_OF_STAR_OR_REGEX`.
> The fix is to build the `NamedStreamingRelation` directly over `RelationChanges`, which is what the non-`CHANGES` branch and `DataStreamReader.changes()` already do:
> ```
> EventTimeWatermark
> +- SubqueryAlias(c)
>    +- NamedStreamingRelation(src)
>       +- RelationChanges
> ```



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org