You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Shixiong Zhu (Jira)" <ji...@apache.org> on 2022/11/08 05:41:00 UTC

[jira] [Created] (SPARK-41040) Self-union streaming query may fail when using readStream.table

Shixiong Zhu created SPARK-41040:
------------------------------------

             Summary: Self-union streaming query may fail when using readStream.table
                 Key: SPARK-41040
                 URL: https://issues.apache.org/jira/browse/SPARK-41040
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 3.4.0
            Reporter: Shixiong Zhu


In a self-union query, the batch plan created by the source will be shared by multiple nodes in the plan. When we transform the plan, the batch plan will be visited multiple times. Hence, the first visit will set the CatalogTable and the second visit will try to set it again and fail the query at [this line|https://github.com/apache/spark/blob/406d0e243cfec9b29df946e1a0e20ed5fe25e152/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala#L625].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org