You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/06 01:45:49 UTC
[GitHub] [spark] brkyvz opened a new pull request #27822:
[SPARK-31061][SQL][3.0] Provide ability to alter the provider of a table
brkyvz opened a new pull request #27822: [SPARK-31061][SQL][3.0] Provide ability to alter the provider of a table
URL: https://github.com/apache/spark/pull/27822
### What changes were proposed in this pull request?
This PR adds functionality to HiveExternalCatalog to be able to change the provider of a table.
### Why are the changes needed?
This is useful for catalogs in Spark 3.0 to be able to use alterTable to change the provider of a table as part of an atomic REPLACE TABLE function.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Unit tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27822: [SPARK-31061][SQL]
Provide ability to alter the provider of a table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27822: [SPARK-31061][SQL] Provide ability to alter the provider of a table
URL: https://github.com/apache/spark/pull/27822#issuecomment-595598186
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27822:
[SPARK-31061][SQL] Provide ability to alter the provider of a table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27822: [SPARK-31061][SQL] Provide ability to alter the provider of a table
URL: https://github.com/apache/spark/pull/27822#issuecomment-595557201
**[Test build #119438 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119438/testReport)** for PR 27822 at commit [`7a5f2c2`](https://github.com/apache/spark/commit/7a5f2c237bd04ab2adf38b3bea841f988a68d25d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27822: [SPARK-31061][SQL]
Provide ability to alter the provider of a table
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27822: [SPARK-31061][SQL] Provide ability to alter the provider of a table
URL: https://github.com/apache/spark/pull/27822#issuecomment-595594506
Hi, @brkyvz . `[3.0]` is used when you make a PR against `branch-3.0` directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27822:
[SPARK-31061][SQL][3.0] Provide ability to alter the provider of a table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27822: [SPARK-31061][SQL][3.0] Provide ability to alter the provider of a table
URL: https://github.com/apache/spark/pull/27822#issuecomment-595557579
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27822: [SPARK-31061][SQL]
Provide ability to alter the provider of a table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27822: [SPARK-31061][SQL] Provide ability to alter the provider of a table
URL: https://github.com/apache/spark/pull/27822#issuecomment-595597850
**[Test build #119438 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119438/testReport)** for PR 27822 at commit [`7a5f2c2`](https://github.com/apache/spark/commit/7a5f2c237bd04ab2adf38b3bea841f988a68d25d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27822:
[SPARK-31061][SQL][3.0] Provide ability to alter the provider of a table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27822: [SPARK-31061][SQL][3.0] Provide ability to alter the provider of a table
URL: https://github.com/apache/spark/pull/27822#issuecomment-595557584
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24175/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27822:
[SPARK-31061][SQL][3.0] Provide ability to alter the provider of a table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27822: [SPARK-31061][SQL][3.0] Provide ability to alter the provider of a table
URL: https://github.com/apache/spark/pull/27822#issuecomment-595557579
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27822:
[SPARK-31061][SQL] Provide ability to alter the provider of a table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27822: [SPARK-31061][SQL] Provide ability to alter the provider of a table
URL: https://github.com/apache/spark/pull/27822#issuecomment-595598189
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119438/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27822:
[SPARK-31061][SQL][3.0] Provide ability to alter the provider of a table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27822: [SPARK-31061][SQL][3.0] Provide ability to alter the provider of a table
URL: https://github.com/apache/spark/pull/27822#issuecomment-595557584
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24175/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27822:
[SPARK-31061][SQL][3.0] Provide ability to alter the provider of a table
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27822: [SPARK-31061][SQL][3.0] Provide ability to alter the provider of a table
URL: https://github.com/apache/spark/pull/27822#discussion_r388695641
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogSuite.scala
##########
@@ -178,4 +180,42 @@ class HiveExternalCatalogSuite extends ExternalCatalogSuite {
assertThrows[QueryExecutionException](client.runSqlHive(
"INSERT overwrite directory \"fs://localhost/tmp\" select 1 as a"))
}
+
+ test("SPARK-31061: alterTable should be able to change table provider") {
+ val catalog = newBasicCatalog()
+ val parquetTable = CatalogTable(
+ identifier = TableIdentifier("parq_tbl", Some("db1")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storageFormat.copy(locationUri = Some(new URI("file:/some/path"))),
+ schema = new StructType().add("col1", "int").add("col2", "string"),
+ provider = Some("parquet"))
+ catalog.createTable(parquetTable, ignoreIfExists = false)
+
+ val rawTable = externalCatalog.getTable("db1", "parq_tbl")
+ assert(rawTable.provider === Some("parquet"))
+
+ val fooTable = parquetTable.copy(provider = Some("foo"))
+ catalog.alterTable(fooTable)
+ val alteredTable = externalCatalog.getTable("db1", "parq_tbl")
+ assert(alteredTable.provider === Some("foo"))
+ }
+
+ test("SPARK-31061: alterTable should be able to change table provider from hive") {
+ val catalog = newBasicCatalog()
+ val parquetTable = CatalogTable(
Review comment:
nit: `hiveTable`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27822:
[SPARK-31061][SQL] Provide ability to alter the provider of a table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27822: [SPARK-31061][SQL] Provide ability to alter the provider of a table
URL: https://github.com/apache/spark/pull/27822#issuecomment-595598186
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27822: [SPARK-31061][SQL]
Provide ability to alter the provider of a table
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27822: [SPARK-31061][SQL] Provide ability to alter the provider of a table
URL: https://github.com/apache/spark/pull/27822
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27822: [SPARK-31061][SQL]
Provide ability to alter the provider of a table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27822: [SPARK-31061][SQL] Provide ability to alter the provider of a table
URL: https://github.com/apache/spark/pull/27822#issuecomment-595598189
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119438/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27822: [SPARK-31061][SQL][3.0]
Provide ability to alter the provider of a table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27822: [SPARK-31061][SQL][3.0] Provide ability to alter the provider of a table
URL: https://github.com/apache/spark/pull/27822#issuecomment-595557201
**[Test build #119438 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119438/testReport)** for PR 27822 at commit [`7a5f2c2`](https://github.com/apache/spark/commit/7a5f2c237bd04ab2adf38b3bea841f988a68d25d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27822: [SPARK-31061][SQL]
Provide ability to alter the provider of a table
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27822: [SPARK-31061][SQL] Provide ability to alter the provider of a table
URL: https://github.com/apache/spark/pull/27822#issuecomment-595643358
thanks, merging to master/3.0! (I've fixed the typo in the test while merging)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org