You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "anchovYu (via GitHub)" <gi...@apache.org> on 2024/01/05 04:07:48 UTC
[PR] [WIP][SQL] Propagate `allowExisting` in view creation when the view/table does not exists [spark]
anchovYu opened a new pull request, #44603:
URL: https://github.com/apache/spark/pull/44603
### What changes were proposed in this pull request?
This PR fixes the undesired behavior that concurrent `CREATE VIEW IF NOT EXISTS` queries could throw `TABLE_OR_VIEW_ALREADY_EXISTS` exceptions. It's because the current implementation did not propagate the 'IF NOT EXISTS' when the detecting view/table does not exists.
### Why are the changes needed?
Fix the above issue.
### Does this PR introduce _any_ user-facing change?
Yes in the sense that if fixes an issue in concurrent case.
### How was this patch tested?
Added a new concurrency test.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46602][SQL] Propagate `allowExisting` in view creation when the view/table does not exists [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #44603:
URL: https://github.com/apache/spark/pull/44603#issuecomment-1878806388
The failed kafka test is unrelated, thanks, merging to master/3.5!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46602][SQL] Propagate `allowExisting` in view creation when the view/table does not exists [spark]
Posted by "anchovYu (via GitHub)" <gi...@apache.org>.
anchovYu commented on code in PR #44603:
URL: https://github.com/apache/spark/pull/44603#discussion_r1442533613
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala:
##########
@@ -153,6 +158,36 @@ abstract class SQLViewSuite extends QueryTest with SQLTestUtils {
}
}
+
+ test("CREATE VIEW IF NOT EXISTS never throws TABLE_OR_VIEW_ALREADY_EXISTS") {
+ // Concurrently create a view with the same name, so that some of the queries may all
+ // get that the view does not exist and try to create it. But with IF NOT EXISTS, the
+ // queries should not fail.
+ import ExecutionContext.Implicits.global
+ val concurrency = 10
+ val tableName = "table_name"
+ val viewName = "view_name"
+ withTable(tableName) {
+ sql(s"CREATE TABLE $tableName (id int) USING parquet")
+ withView("view_name") {
+ val futures = (0 to concurrency).map { _ =>
Review Comment:
Yes I have verified that without the fix this test failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46602][SQL] Propagate `allowExisting` in view creation when the view/table does not exists [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #44603: [SPARK-46602][SQL] Propagate `allowExisting` in view creation when the view/table does not exists
URL: https://github.com/apache/spark/pull/44603
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46602][SQL] Propagate `allowExisting` in view creation when the view/table does not exists [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #44603:
URL: https://github.com/apache/spark/pull/44603#discussion_r1442515458
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala:
##########
@@ -153,6 +158,36 @@ abstract class SQLViewSuite extends QueryTest with SQLTestUtils {
}
}
+
+ test("CREATE VIEW IF NOT EXISTS never throws TABLE_OR_VIEW_ALREADY_EXISTS") {
+ // Concurrently create a view with the same name, so that some of the queries may all
+ // get that the view does not exist and try to create it. But with IF NOT EXISTS, the
+ // queries should not fail.
+ import ExecutionContext.Implicits.global
+ val concurrency = 10
+ val tableName = "table_name"
+ val viewName = "view_name"
+ withTable(tableName) {
+ sql(s"CREATE TABLE $tableName (id int) USING parquet")
+ withView("view_name") {
+ val futures = (0 to concurrency).map { _ =>
Review Comment:
the fix is trivial, we don't need to commit this fragile and heavy test. You can just run it off line and verify that the fix works.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala:
##########
@@ -153,6 +158,36 @@ abstract class SQLViewSuite extends QueryTest with SQLTestUtils {
}
}
+
+ test("CREATE VIEW IF NOT EXISTS never throws TABLE_OR_VIEW_ALREADY_EXISTS") {
+ // Concurrently create a view with the same name, so that some of the queries may all
+ // get that the view does not exist and try to create it. But with IF NOT EXISTS, the
+ // queries should not fail.
+ import ExecutionContext.Implicits.global
+ val concurrency = 10
+ val tableName = "table_name"
+ val viewName = "view_name"
+ withTable(tableName) {
+ sql(s"CREATE TABLE $tableName (id int) USING parquet")
+ withView("view_name") {
+ val futures = (0 to concurrency).map { _ =>
Review Comment:
the fix is trivial, we don't need to commit this fragile and heavy test. You can just run it offline and verify that the fix works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org