You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ouyangxiaochen <gi...@git.apache.org> on 2017/01/19 03:20:36 UTC
[GitHub] spark pull request #16638: spark-19115
GitHub user ouyangxiaochen opened a pull request:
https://github.com/apache/spark/pull/16638
spark-19115
## What changes were proposed in this pull request?
sparksql supports the command : create external table if not exists gen_tbl like src_tbl location '/warehouse/gen_tbl' in spark2.X
## How was this patch tested?
manual tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ouyangxiaochen/spark spark19115
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16638.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16638
----
commit adde008588cf8e05cf261c086201c27a8dd5584f
Author: ouyangxiaochen <ou...@zte.com.cn>
Date: 2017-01-19T03:15:17Z
spark-19115
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16638
**[Test build #72550 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72550/testReport)** for PR 16638 at commit [`71f1d12`](https://github.com/apache/spark/commit/71f1d1202e8f95194499e4aa5168fff08cc40427).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by ouyangxiaochen <gi...@git.apache.org>.
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
I have fixed the error of test cases and they run successfully. So,please run the test cases again.Thanks a lot! @SparkQA
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16638
please address https://github.com/apache/spark/pull/16638#discussion_r97471913
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by ouyangxiaochen <gi...@git.apache.org>.
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
Happy Chinese New Year ! @gatorsmile
The Spring Festival holiday just ended, and I return to work today, what work do I need to do?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: [SPARK-19115] [SQL] Supporting Create External Ta...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r97469240
--- Diff: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ---
@@ -81,8 +81,8 @@ statement
rowFormat? createFileFormat? locationSpec?
(TBLPROPERTIES tablePropertyList)?
(AS? query)? #createHiveTable
- | CREATE TABLE (IF NOT EXISTS)? target=tableIdentifier
- LIKE source=tableIdentifier #createTableLike
+ | CREATE EXTERNAL? TABLE (IF NOT EXISTS)? target=tableIdentifier
--- End diff --
since Spark 2.2, we wanna hide the manage/external concept from users. It looks reasonable to add a `LOCATION` statement in `CREATE TABLE LIKE`, but do we really need the `EXTERNAL` keyword? We don't need to be exactly same with hive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16638
**[Test build #71904 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71904/testReport)** for PR 16638 at commit [`b80f8e6`](https://github.com/apache/spark/commit/b80f8e66e1cbb7111c090358cabc925c6af233d2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by ouyangxiaochen <gi...@git.apache.org>.
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
I am sorry that I did't grasp the key points of your question. In Hive, if there are data files under the specified path while creating an external table, then Hive will identify the files as table data files.
In many spark applications, external table data is generated by other applications under the external table path. So, Hive did nothing with the directory specified in the LOCATION.
Thank you for your patience and guidance. @gatorsmile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16638
please resolve the conflict too, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: spark-19115
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16638
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by ouyangxiaochen <gi...@git.apache.org>.
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
Oh, I See, I miss a step \u2019git remote add upstream ...\u2018.
But now, I have delete my repository in my profile. So this PR can\u2018t know which repository should be associated. So, do u have a method to help me cover this problem?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: [SPARK-19115] [SQL] Supporting Create External Ta...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r99978824
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala ---
@@ -528,8 +528,8 @@ class HiveDDLCommandSuite extends PlanTest with SQLTestUtils with TestHiveSingle
assert(source.table == "table2")
val v2 = "CREATE TABLE IF NOT EXISTS table1 LIKE table2"
--- End diff --
add one more test case to check CREATE TABLE LIKE with location
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16638
**[Test build #71904 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71904/testReport)** for PR 16638 at commit [`b80f8e6`](https://github.com/apache/spark/commit/b80f8e66e1cbb7111c090358cabc925c6af233d2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: [SPARK-19115] [SQL] Supporting Create External Ta...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r99988384
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -812,150 +812,234 @@ class HiveDDLSuite
}
}
- test("CREATE TABLE LIKE a temporary view") {
- val sourceViewName = "tab1"
- val targetTabName = "tab2"
- withTempView(sourceViewName) {
- withTable(targetTabName) {
- spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
- .createTempView(sourceViewName)
- sql(s"CREATE TABLE $targetTabName LIKE $sourceViewName")
-
- val sourceTable = spark.sessionState.catalog.getTempViewOrPermanentTableMetadata(
- TableIdentifier(sourceViewName))
- val targetTable = spark.sessionState.catalog.getTableMetadata(
- TableIdentifier(targetTabName, Some("default")))
-
- checkCreateTableLike(sourceTable, targetTable)
+ test("CREATE TABLE LIKE a temporary view [LOCATION]...") {
+ var createdTableType = "MANAGED"
+ for ( i <- 0 to 1 ) {
--- End diff --
creating a method and wrap this piece of code can also reuse the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by ouyangxiaochen <gi...@git.apache.org>.
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
Should I delete my local master repository firstly\uff0cand fork a new one again? @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: [SPARK-19115] [SQL] Supporting Create External Ta...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r99979027
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -812,150 +812,234 @@ class HiveDDLSuite
}
}
- test("CREATE TABLE LIKE a temporary view") {
- val sourceViewName = "tab1"
- val targetTabName = "tab2"
- withTempView(sourceViewName) {
- withTable(targetTabName) {
- spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
- .createTempView(sourceViewName)
- sql(s"CREATE TABLE $targetTabName LIKE $sourceViewName")
-
- val sourceTable = spark.sessionState.catalog.getTempViewOrPermanentTableMetadata(
- TableIdentifier(sourceViewName))
- val targetTable = spark.sessionState.catalog.getTableMetadata(
- TableIdentifier(targetTabName, Some("default")))
-
- checkCreateTableLike(sourceTable, targetTable)
+ test("CREATE TABLE LIKE a temporary view [LOCATION]...") {
+ var createdTableType = "MANAGED"
+ for ( i <- 0 to 1 ) {
--- End diff --
you can create a method with parameter `location: Option[String]`, instead of writing a for loop with 2 iterations...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16638
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72576/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16638
Your master is clean (i.e., exactly identical to the upstream/master), right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16638
You might be able to make it by forcefully pushing the new changes by `git push -f origin NEW_BRANCH:REMOTE_BRANCH `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16638
**[Test build #72550 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72550/testReport)** for PR 16638 at commit [`71f1d12`](https://github.com/apache/spark/commit/71f1d1202e8f95194499e4aa5168fff08cc40427).
* This patch passes all tests.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: [SPARK-19115] [SQL] Supporting Create External Ta...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r99978630
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -51,13 +51,14 @@ import org.apache.spark.util.Utils
*
* The syntax of using this command in SQL is:
* {{{
- * CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
- * LIKE [other_db_name.]existing_table_name
+ * CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
--- End diff --
no `EXTERNAL`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: spark-19115
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16638
We have a few test cases you can follow. Please create test cases. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71904/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16638
@ouyangxiaochen Please do not duplicate the test cases. Try to combine them.
@cloud-fan @yhuai Could you please check whether such a DDL support is desirable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16638
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: spark-19115
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16638
First, please change the PR title to `[SPARK-19115] [SQL] Supporting Create External Table Like Location`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: spark-19115
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r96809883
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -58,6 +58,7 @@ import org.apache.spark.util.Utils
case class CreateTableLikeCommand(
--- End diff --
Please update the comment of this class.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by ouyangxiaochen <gi...@git.apache.org>.
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
I met some troubles when I resolving the conflict, So can u give me some guidances? Thanks a lot! @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by ouyangxiaochen <gi...@git.apache.org>.
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
My master branch with the master of Apache is not synchronized, and then I did the pull operation, my master branch still not synchronized, and finally I removed my remote repository.
But I do not know how to associate a new branch with this PR? I Think I made a misopreation. @gatorsmile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: [SPARK-19115] [SQL] Supporting Create External Ta...
Posted by ouyangxiaochen <gi...@git.apache.org>.
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r99984076
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -812,150 +812,234 @@ class HiveDDLSuite
}
}
- test("CREATE TABLE LIKE a temporary view") {
- val sourceViewName = "tab1"
- val targetTabName = "tab2"
- withTempView(sourceViewName) {
- withTable(targetTabName) {
- spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
- .createTempView(sourceViewName)
- sql(s"CREATE TABLE $targetTabName LIKE $sourceViewName")
-
- val sourceTable = spark.sessionState.catalog.getTempViewOrPermanentTableMetadata(
- TableIdentifier(sourceViewName))
- val targetTable = spark.sessionState.catalog.getTableMetadata(
- TableIdentifier(targetTabName, Some("default")))
-
- checkCreateTableLike(sourceTable, targetTable)
+ test("CREATE TABLE LIKE a temporary view [LOCATION]...") {
+ var createdTableType = "MANAGED"
+ for ( i <- 0 to 1 ) {
--- End diff --
I write this for the purpose of reusing this piece of public code, because the basic logic of these two scenarios are almost the same.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16638
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by ouyangxiaochen <gi...@git.apache.org>.
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
Here's how I create a PR:
1.fork the master of Apache;
2.create a new branch in my master branch
3.select my new branch menu and create a new PR.
4.edit my new branch code.
5.commit and push.
Can u point lost or mistake steps for me, Thank u for your guidances\uff01 @gatorsmile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16638
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71887/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: [SPARK-19115] [SQL] Supporting Create External Ta...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r97446544
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -1140,14 +1140,18 @@ class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder {
*
* For example:
* {{{
- * CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
- * LIKE [other_db_name.]existing_table_name
+ * CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
+ * LIKE [other_db_name.]existing_table_name [locationSpec]
* }}}
*/
override def visitCreateTableLike(ctx: CreateTableLikeContext): LogicalPlan = withOrigin(ctx) {
val targetTable = visitTableIdentifier(ctx.target)
val sourceTable = visitTableIdentifier(ctx.source)
- CreateTableLikeCommand(targetTable, sourceTable, ctx.EXISTS != null)
+ val location = Option(ctx.locationSpec).map(visitLocationSpec)
+ if (ctx.EXTERNAL != null && location.isEmpty) {
+ operationNotAllowed("CREATE EXTERNAL TABLE LIKE must be accompanied by LOCATION", ctx)
+ }
--- End diff --
To the other reviewers, we are following what we did in `visitCreateHiveTable`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16638
ok to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: [SPARK-19115] [SQL] Supporting Create External Ta...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r99979122
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -812,150 +812,234 @@ class HiveDDLSuite
}
}
- test("CREATE TABLE LIKE a temporary view") {
- val sourceViewName = "tab1"
- val targetTabName = "tab2"
- withTempView(sourceViewName) {
- withTable(targetTabName) {
- spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
- .createTempView(sourceViewName)
- sql(s"CREATE TABLE $targetTabName LIKE $sourceViewName")
-
- val sourceTable = spark.sessionState.catalog.getTempViewOrPermanentTableMetadata(
- TableIdentifier(sourceViewName))
- val targetTable = spark.sessionState.catalog.getTableMetadata(
- TableIdentifier(targetTabName, Some("default")))
-
- checkCreateTableLike(sourceTable, targetTable)
+ test("CREATE TABLE LIKE a temporary view [LOCATION]...") {
+ var createdTableType = "MANAGED"
+ for ( i <- 0 to 1 ) {
+ withTempDir {tmpDir =>
+ val sourceViewName = "tab1"
+ val targetTabName = "tab2"
+ val basePath = tmpDir.toURI
+ withTempView(sourceViewName) {
+ withTable(targetTabName) {
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+ .createTempView(sourceViewName)
+ if (i == 0) {
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceViewName ")
+ } else {
+ createdTableType = "EXTERNAL"
+ sql(s"CREATE TABLE $targetTabName " +
+ s"LIKE $sourceViewName LOCATION '$basePath'")
+ }
+
+ val sourceTable = spark.sessionState.catalog.getTempViewOrPermanentTableMetadata(
+ TableIdentifier(sourceViewName))
+ val targetTable = spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(targetTabName, Some("default")))
+
+ checkCreateTableLike(sourceTable, targetTable, createdTableType)
+ }
+ }
}
}
}
- test("CREATE TABLE LIKE a data source table") {
- val sourceTabName = "tab1"
- val targetTabName = "tab2"
- withTable(sourceTabName, targetTabName) {
- spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
- .write.format("json").saveAsTable(sourceTabName)
- sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
-
- val sourceTable =
- spark.sessionState.catalog.getTableMetadata(TableIdentifier(sourceTabName, Some("default")))
- val targetTable =
- spark.sessionState.catalog.getTableMetadata(TableIdentifier(targetTabName, Some("default")))
- // The table type of the source table should be a Hive-managed data source table
- assert(DDLUtils.isDatasourceTable(sourceTable))
- assert(sourceTable.tableType == CatalogTableType.MANAGED)
-
- checkCreateTableLike(sourceTable, targetTable)
+ test("CREATE TABLE LIKE a data source table [LOCATION]...") {
+ var createdTableType = "MANAGED"
+ for ( i <- 0 to 1 ) {
+ withTempDir { tmpDir =>
+ val sourceTabName = "tab1"
+ val targetTabName = "tab2"
+ val basePath = tmpDir.toURI
+ withTable(sourceTabName, targetTabName) {
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+ .write.format("json").saveAsTable(sourceTabName)
+ if ( i == 0 ) {
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+ } else {
+ createdTableType = "EXTERNAL"
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName LOCATION '$basePath'")
+ }
+
+ val sourceTable =
+ spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(sourceTabName, Some("default")))
+ val targetTable =
+ spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(targetTabName, Some("default")))
+ // The table type of the source table should be a Hive-managed data source table
+ assert(DDLUtils.isDatasourceTable(sourceTable))
+ assert(sourceTable.tableType == CatalogTableType.MANAGED)
+
+ checkCreateTableLike(sourceTable, targetTable, createdTableType)
+ }
+ }
}
}
- test("CREATE TABLE LIKE an external data source table") {
- val sourceTabName = "tab1"
- val targetTabName = "tab2"
- withTable(sourceTabName, targetTabName) {
- withTempPath { dir =>
- val path = dir.getCanonicalPath
- spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
- .write.format("parquet").save(path)
- sql(s"CREATE TABLE $sourceTabName USING parquet OPTIONS (PATH '${dir.toURI}')")
- sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
-
- // The source table should be an external data source table
- val sourceTable = spark.sessionState.catalog.getTableMetadata(
- TableIdentifier(sourceTabName, Some("default")))
- val targetTable = spark.sessionState.catalog.getTableMetadata(
- TableIdentifier(targetTabName, Some("default")))
- // The table type of the source table should be an external data source table
- assert(DDLUtils.isDatasourceTable(sourceTable))
- assert(sourceTable.tableType == CatalogTableType.EXTERNAL)
-
- checkCreateTableLike(sourceTable, targetTable)
+ test("CREATE TABLE LIKE an external data source table [LOCATION]...") {
+ var createdTableType = "MANAGED"
+ for ( i <- 0 to 1 ) {
+ withTempDir { tmpDir =>
+ val sourceTabName = "tab1"
+ val targetTabName = "tab2"
+ val basePath = tmpDir.toURI
+ withTable(sourceTabName, targetTabName) {
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+ .write.format("parquet").save(path)
+ sql(s"CREATE TABLE $sourceTabName USING parquet OPTIONS (PATH '${dir.toURI}')")
+ if ( i == 0 ) {
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+ } else {
+ createdTableType = "EXTERNAL"
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName LOCATION '$basePath'")
+ }
+
+ // The source table should be an external data source table
+ val sourceTable = spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(sourceTabName, Some("default")))
+ val targetTable = spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(targetTabName, Some("default")))
+ // The table type of the source table should be an external data source table
+ assert(DDLUtils.isDatasourceTable(sourceTable))
+ assert(sourceTable.tableType == CatalogTableType.EXTERNAL)
+
+ checkCreateTableLike(sourceTable, targetTable, createdTableType)
+ }
+ }
}
}
}
- test("CREATE TABLE LIKE a managed Hive serde table") {
- val catalog = spark.sessionState.catalog
- val sourceTabName = "tab1"
- val targetTabName = "tab2"
- withTable(sourceTabName, targetTabName) {
- sql(s"CREATE TABLE $sourceTabName TBLPROPERTIES('prop1'='value1') AS SELECT 1 key, 'a'")
- sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
-
- val sourceTable = catalog.getTableMetadata(TableIdentifier(sourceTabName, Some("default")))
- assert(sourceTable.tableType == CatalogTableType.MANAGED)
- assert(sourceTable.properties.get("prop1").nonEmpty)
- val targetTable = catalog.getTableMetadata(TableIdentifier(targetTabName, Some("default")))
-
- checkCreateTableLike(sourceTable, targetTable)
+ test("CREATE TABLE LIKE a managed Hive serde table [LOCATION]...") {
+ var createdTableType = "MANAGED"
+ for ( i <- 0 to 1 ) {
+ val catalog = spark.sessionState.catalog
+ withTempDir { tmpDir =>
+ val sourceTabName = "tab1"
+ val targetTabName = "tab2"
+ val basePath = tmpDir.toURI
+ withTable(sourceTabName, targetTabName) {
+ sql(s"CREATE TABLE $sourceTabName TBLPROPERTIES('prop1'='value1') AS SELECT 1 key, 'a'")
+
+ if ( i == 0 ) {
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+ } else {
+ createdTableType = "EXTERNAL"
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName LOCATION '$basePath'")
+ }
+
+ val sourceTable = catalog.getTableMetadata(
+ TableIdentifier(sourceTabName, Some("default")))
+ assert(sourceTable.tableType == CatalogTableType.MANAGED)
+ assert(sourceTable.properties.get("prop1").nonEmpty)
+ val targetTable = catalog.getTableMetadata(
+ TableIdentifier(targetTabName, Some("default")))
+
+ checkCreateTableLike(sourceTable, targetTable, createdTableType)
+ }
+ }
}
}
- test("CREATE TABLE LIKE an external Hive serde table") {
- val catalog = spark.sessionState.catalog
- withTempDir { tmpDir =>
- val basePath = tmpDir.toURI
- val sourceTabName = "tab1"
- val targetTabName = "tab2"
- withTable(sourceTabName, targetTabName) {
- assert(tmpDir.listFiles.isEmpty)
- sql(
- s"""
- |CREATE EXTERNAL TABLE $sourceTabName (key INT comment 'test', value STRING)
- |COMMENT 'Apache Spark'
- |PARTITIONED BY (ds STRING, hr STRING)
- |LOCATION '$basePath'
- """.stripMargin)
- for (ds <- Seq("2008-04-08", "2008-04-09"); hr <- Seq("11", "12")) {
- sql(
- s"""
- |INSERT OVERWRITE TABLE $sourceTabName
- |partition (ds='$ds',hr='$hr')
- |SELECT 1, 'a'
+ test("CREATE TABLE LIKE an external Hive serde table [LOCATION]...") {
+ var createdTableType = "MANAGED"
+ for ( i <- 0 to 1 ) {
+ val catalog = spark.sessionState.catalog
+ withTempDir { tmpDir =>
+ val basePath = tmpDir.toURI
+ withTempDir { tmpDir1 =>
+ val basePath1 = tmpDir1.toURI
+ val sourceTabName = "tab1"
+ val targetTabName = "tab2"
+ withTable(sourceTabName, targetTabName) {
+ assert(tmpDir.listFiles.isEmpty)
+ sql(
+ s"""
+ |CREATE EXTERNAL TABLE $sourceTabName (key INT comment 'test', value STRING)
+ |COMMENT 'Apache Spark'
+ |PARTITIONED BY (ds STRING, hr STRING)
+ |LOCATION '$basePath'
""".stripMargin)
+ for (ds <- Seq("2008-04-08", "2008-04-09"); hr <- Seq("11", "12")) {
+ sql(
+ s"""
+ |INSERT OVERWRITE TABLE $sourceTabName
+ |partition (ds='$ds',hr='$hr')
+ |SELECT 1, 'a'
+ """.stripMargin)
+ }
+
+ if ( i == 0 ) {
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+ } else {
+ createdTableType = "EXTERNAL"
+ sql(s"CREATE TABLE $targetTabName " +
+ s"LIKE $sourceTabName LOCATION '$basePath1'")
+ }
+
+ val sourceTable = catalog.getTableMetadata(
+ TableIdentifier(sourceTabName, Some("default")))
+ assert(sourceTable.tableType == CatalogTableType.EXTERNAL)
+ assert(sourceTable.comment == Option("Apache Spark"))
+ val targetTable = catalog.getTableMetadata(
+ TableIdentifier(targetTabName, Some("default")))
+
+ checkCreateTableLike(sourceTable, targetTable, createdTableType)
+ }
}
- sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
-
- val sourceTable = catalog.getTableMetadata(TableIdentifier(sourceTabName, Some("default")))
- assert(sourceTable.tableType == CatalogTableType.EXTERNAL)
- assert(sourceTable.comment == Option("Apache Spark"))
- val targetTable = catalog.getTableMetadata(TableIdentifier(targetTabName, Some("default")))
-
- checkCreateTableLike(sourceTable, targetTable)
}
}
+
}
- test("CREATE TABLE LIKE a view") {
- val sourceTabName = "tab1"
- val sourceViewName = "view"
- val targetTabName = "tab2"
- withTable(sourceTabName, targetTabName) {
- withView(sourceViewName) {
- spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
- .write.format("json").saveAsTable(sourceTabName)
- sql(s"CREATE VIEW $sourceViewName AS SELECT * FROM $sourceTabName")
- sql(s"CREATE TABLE $targetTabName LIKE $sourceViewName")
-
- val sourceView = spark.sessionState.catalog.getTableMetadata(
- TableIdentifier(sourceViewName, Some("default")))
- // The original source should be a VIEW with an empty path
- assert(sourceView.tableType == CatalogTableType.VIEW)
- assert(sourceView.viewText.nonEmpty && sourceView.viewOriginalText.nonEmpty)
- val targetTable = spark.sessionState.catalog.getTableMetadata(
- TableIdentifier(targetTabName, Some("default")))
-
- checkCreateTableLike(sourceView, targetTable)
+ test("CREATE TABLE LIKE a view [LOCATION]...") {
+ var createdTableType = "MANAGED"
+ for ( i <- 0 to 1 ) {
+ withTempDir { tmpDir =>
+ val sourceTabName = "tab1"
+ val sourceViewName = "view"
+ val targetTabName = "tab2"
+ val basePath = tmpDir.toURI
+ withTable(sourceTabName, targetTabName) {
+ withView(sourceViewName) {
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+ .write.format("json").saveAsTable(sourceTabName)
+ sql(s"CREATE VIEW $sourceViewName AS SELECT * FROM $sourceTabName")
+
+ if ( i == 0 ) {
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceViewName")
+ } else {
+ createdTableType = "EXTERNAL"
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceViewName LOCATION '$basePath'")
+ }
+
+ val sourceView = spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(sourceViewName, Some("default")))
+ // The original source should be a VIEW with an empty path
+ assert(sourceView.tableType == CatalogTableType.VIEW)
+ assert(sourceView.viewText.nonEmpty && sourceView.viewOriginalText.nonEmpty)
+ val targetTable = spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(targetTabName, Some("default")))
+
+ checkCreateTableLike(sourceView, targetTable, createdTableType)
+ }
+ }
}
}
+
}
- private def checkCreateTableLike(sourceTable: CatalogTable, targetTable: CatalogTable): Unit = {
- // The created table should be a MANAGED table with empty view text and original text.
- assert(targetTable.tableType == CatalogTableType.MANAGED,
- "the created table must be a Hive managed table")
+ private def checkCreateTableLike(
+ sourceTable: CatalogTable,
+ targetTable: CatalogTable,
+ tableType: String): Unit = {
--- End diff --
why not pass in a `CataogTableType` instead of a string?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16638
You do not need to do the step 1 every time. You might miss the following two steps when you want to resolve your conflicts.
> git fetch upstream
> git merge upstream/master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: spark-19115
Posted by ouyangxiaochen <gi...@git.apache.org>.
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r96993195
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -58,6 +58,7 @@ import org.apache.spark.util.Utils
case class CreateTableLikeCommand(
--- End diff --
ok,i will update it later,Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: [SPARK-19115] [SQL] Supporting Create External Ta...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/16638
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: [SPARK-19115] [SQL] Supporting Create External Ta...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r99978757
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala ---
@@ -518,8 +518,8 @@ class HiveDDLCommandSuite extends PlanTest with SQLTestUtils with TestHiveSingle
test("create table like") {
val v1 = "CREATE TABLE table1 LIKE table2"
- val (target, source, exists) = parser.parsePlan(v1).collect {
- case CreateTableLikeCommand(t, s, allowExisting) => (t, s, allowExisting)
+ val (target, source, location, exists) = parser.parsePlan(v1).collect {
--- End diff --
add an assert to check `location` is empty
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: spark-19115
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16638
Let me rephrase it. If the directory specified in the `LOCATION` spec contains the other files, what does Hive behave?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16638
**[Test build #71887 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71887/testReport)** for PR 16638 at commit [`713ca97`](https://github.com/apache/spark/commit/713ca9752a7e37b9e1d52dc0b3171c9c3888ab19).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16638
**[Test build #72576 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72576/testReport)** for PR 16638 at commit [`5dd21b2`](https://github.com/apache/spark/commit/5dd21b28e46b42f2af853b48b014f05d7d6b7895).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by ouyangxiaochen <gi...@git.apache.org>.
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
OK. I'll try it immediately. Thank U very much!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: spark-19115
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16638
Could you follow the title requirement in http://spark.apache.org/contributing.html?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16638
**[Test build #72576 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72576/testReport)** for PR 16638 at commit [`5dd21b2`](https://github.com/apache/spark/commit/5dd21b28e46b42f2af853b48b014f05d7d6b7895).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16638
ping @ouyangxiaochen : )
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by ouyangxiaochen <gi...@git.apache.org>.
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
ping @gatorsmile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: spark-19115
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16638
Please keep updating your PR description. For example, this PR is not relying on `manual tests`. In addition, you also need to summarize what this PR did. List more details to help reviewers understand your changes and impacts. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16638
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72550/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: [SPARK-19115] [SQL] Supporting Create External Ta...
Posted by ouyangxiaochen <gi...@git.apache.org>.
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r97454893
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -1140,14 +1140,18 @@ class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder {
*
* For example:
* {{{
- * CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
- * LIKE [other_db_name.]existing_table_name
+ * CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
+ * LIKE [other_db_name.]existing_table_name [locationSpec]
* }}}
*/
override def visitCreateTableLike(ctx: CreateTableLikeContext): LogicalPlan = withOrigin(ctx) {
val targetTable = visitTableIdentifier(ctx.target)
val sourceTable = visitTableIdentifier(ctx.source)
- CreateTableLikeCommand(targetTable, sourceTable, ctx.EXISTS != null)
+ val location = Option(ctx.locationSpec).map(visitLocationSpec)
+ if (ctx.EXTERNAL != null && location.isEmpty) {
--- End diff --
OK, I'll do it later, Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: [SPARK-19115] [SQL] Supporting Create External Ta...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r97471178
--- Diff: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ---
@@ -81,8 +81,8 @@ statement
rowFormat? createFileFormat? locationSpec?
(TBLPROPERTIES tablePropertyList)?
(AS? query)? #createHiveTable
- | CREATE TABLE (IF NOT EXISTS)? target=tableIdentifier
- LIKE source=tableIdentifier #createTableLike
+ | CREATE EXTERNAL? TABLE (IF NOT EXISTS)? target=tableIdentifier
--- End diff --
I am fine
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16638
you can start with a new branch and apply the changes manually, e.g. copy code from this PR to the new branch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16638
**[Test build #71887 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71887/testReport)** for PR 16638 at commit [`713ca97`](https://github.com/apache/spark/commit/713ca9752a7e37b9e1d52dc0b3171c9c3888ab19).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: spark-19115
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16638
In the PR, you might need to consider more scenarios. For example, let me ask a question. How does Hive behave when the specified location is not empty?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: spark-19115
Posted by ouyangxiaochen <gi...@git.apache.org>.
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
Here is the differences between Hive and Spark2.x as follow:
1.Hive
create table test(id int); --> MANAGED_TABLE
create table test(id int) location '/warehouse/test'; --> MANAGED_TABLE
create external table test(id int) location '/warehouse/test'; --> EXTERNAL_TABLE
create external table test(id int); --> EXTERNAL_TABLE
2.Spark2.x:
create table test(id int); --> MANAGED_TABLE
create table test(id int) location '/warehouse/test'; --> EXTERNAL_TABLE
create external table test(id int) location '/warehouse/test'; --> EXTERNAL_TABLE
create external table test(id int); --> operationNotAllowed("CREATE EXTERNAL TABLE must be accompanied by LOCATION", ctx)
So,this PR follows the spark2.x\u2018s design rules,thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: [SPARK-19115] [SQL] Supporting Create External Ta...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r99978898
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala ---
@@ -812,150 +812,234 @@ class HiveDDLSuite
}
}
- test("CREATE TABLE LIKE a temporary view") {
- val sourceViewName = "tab1"
- val targetTabName = "tab2"
- withTempView(sourceViewName) {
- withTable(targetTabName) {
- spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
- .createTempView(sourceViewName)
- sql(s"CREATE TABLE $targetTabName LIKE $sourceViewName")
-
- val sourceTable = spark.sessionState.catalog.getTempViewOrPermanentTableMetadata(
- TableIdentifier(sourceViewName))
- val targetTable = spark.sessionState.catalog.getTableMetadata(
- TableIdentifier(targetTabName, Some("default")))
-
- checkCreateTableLike(sourceTable, targetTable)
+ test("CREATE TABLE LIKE a temporary view [LOCATION]...") {
--- End diff --
actually we don't need to change the test name
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16638
No worry, open/submit a new PR. : )
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16638
Build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16638
You might not be familiar with the Github/Git. How about submitting a new PR? : )
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: [SPARK-19115] [SQL] Supporting Create External Ta...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r97446466
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -1140,14 +1140,18 @@ class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder {
*
* For example:
* {{{
- * CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
- * LIKE [other_db_name.]existing_table_name
+ * CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
+ * LIKE [other_db_name.]existing_table_name [locationSpec]
* }}}
*/
override def visitCreateTableLike(ctx: CreateTableLikeContext): LogicalPlan = withOrigin(ctx) {
val targetTable = visitTableIdentifier(ctx.target)
val sourceTable = visitTableIdentifier(ctx.source)
- CreateTableLikeCommand(targetTable, sourceTable, ctx.EXISTS != null)
+ val location = Option(ctx.locationSpec).map(visitLocationSpec)
+ if (ctx.EXTERNAL != null && location.isEmpty) {
--- End diff --
Add a comment here:
```
// If we are creating an EXTERNAL table, then the LOCATION field is required
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16638
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16638: [SPARK-19115] [SQL] Supporting Create External Table Lik...
Posted by ouyangxiaochen <gi...@git.apache.org>.
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
I have created a PR at [https://github.com/apache/spark/pull/16868](url), please review it, Thanks! @gatorsmile @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16638: [SPARK-19115] [SQL] Supporting Create External Ta...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r97471913
--- Diff: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ---
@@ -81,8 +81,8 @@ statement
rowFormat? createFileFormat? locationSpec?
(TBLPROPERTIES tablePropertyList)?
(AS? query)? #createHiveTable
- | CREATE TABLE (IF NOT EXISTS)? target=tableIdentifier
- LIKE source=tableIdentifier #createTableLike
+ | CREATE EXTERNAL? TABLE (IF NOT EXISTS)? target=tableIdentifier
--- End diff --
ok then let's simplify the logic: if `location` is specified, we create an external table internally. Else, create managed table.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org