[jira] [Commented] (SPARK-41040) Self-union streaming query may fail when using readStream.table

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-41040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17630203#comment-17630203 ] 

Apache Spark commented on SPARK-41040:
--------------------------------------

User 'zsxwing' has created a pull request for this issue:
https://github.com/apache/spark/pull/38553

> Self-union streaming query may fail when using readStream.table
> ---------------------------------------------------------------
>
>                 Key: SPARK-41040
>                 URL: https://issues.apache.org/jira/browse/SPARK-41040
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 3.4.0
>            Reporter: Shixiong Zhu
>            Assignee: Shixiong Zhu
>            Priority: Major
>
> In a self-union query, the batch plan created by the source will be shared by multiple nodes in the plan. When we transform the plan, the batch plan will be visited multiple times. Hence, the first visit will set the CatalogTable and the second visit will try to set it again and fail the query at [this line|https://github.com/apache/spark/blob/406d0e243cfec9b29df946e1a0e20ed5fe25e152/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala#L625].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org