You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/08 06:14:52 UTC
[GitHub] [spark] zsxwing opened a new pull request, #38553: [SPARK-41040][SS]
zsxwing opened a new pull request, #38553:
URL: https://github.com/apache/spark/pull/38553
### What changes were proposed in this pull request?
https://github.com/apache/spark/pull/36963 added a check to disallow any source setting `CatalogTable` in the batch plan. However, this check is not safe to enforce:
- In a self-union query, the batch plan created by the source will be shared by multiple nodes in the plan. When we transform the plan, the batch plan will be visited multiple times. Hence, the first visit will set the `CatalogTable` and the second visit will try to set it again and fail the query.
- A source built by arbitrary developers can set `CatalogTable` in the batch plan. We should not fail as it would break an existing source.
This PR fixes the issue by removing the check and set `CatalogTable` only if the batch plan doesn't have one.
### Why are the changes needed?
Fix a bug in master.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
The new added unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] asfgit closed pull request #38553: [SPARK-41040][SS] Fix self-union streaming query failure when using readStream.table
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #38553: [SPARK-41040][SS] Fix self-union streaming query failure when using readStream.table
URL: https://github.com/apache/spark/pull/38553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zsxwing commented on pull request #38553: [SPARK-41040][SS] Fix self-union streaming query failure when using readStream.table
Posted by GitBox <gi...@apache.org>.
zsxwing commented on PR #38553:
URL: https://github.com/apache/spark/pull/38553#issuecomment-1307494412
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org