You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/03 14:17:10 UTC
[GitHub] [spark] ulysses-you opened a new pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
ulysses-you opened a new pull request #28647:
URL: https://github.com/apache/spark/pull/28647
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
At CreateTableLikeCommand, we use the new tblproperties with merge source tblproperties.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
We should retain the useful tblproperties, e.g. `parquet.compression`. And hive also retain the tblproperties.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Add UT.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-720790697
thank you @dongjoon-hyun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r448055181
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##########
@@ -115,6 +116,26 @@ case class CreateTableLikeCommand(
CatalogTableType.EXTERNAL
}
+ // We only copy source tbl properties if the format is the same with each other
+ val needCopyProperties =
+ (provider.isEmpty || provider == sourceTableDesc.provider) &&
Review comment:
`newProvider` checked `VIEW` and always used `parquet` as the provider.
We only copy properties for tables, and already checked at below code
```
val newProperties = sourceTableDesc.tableType match {
case MANAGED | EXTERNAL if needCopyProperties =>
// Hive only retain the useful properties through serde class annotation.
// For better compatible with Hive, we remove the metastore properties.
sourceTableDesc.properties -- DDLUtils.METASTORE_GENERATED_PROPERTIES ++ properties
case MANAGED | EXTERNAL =>
properties
case VIEW =>
// For view, we just use new properties
properties
case other =>
throw new IllegalArgumentException(
s"Unknown table type is found at createTableLikeCommand: $other")
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652124188
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-655221228
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-651164295
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r433668494
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -839,6 +839,19 @@ case class AlterTableSetLocationCommand(
object DDLUtils {
val HIVE_PROVIDER = "hive"
+ val METASTORE_GENERATED_PROPERTIES: Set[String] = Set(
+ "CreateTime",
+ "transient_lastDdlTime",
+ "grantTime",
+ "lastUpdateTime",
+ "last_modified_by",
+ "last_modified_time",
+ "Owner:",
+ "totalNumberFiles",
+ "maxFileSize",
+ "minFileSize"
Review comment:
> We need to add some properties like last_modified_by , last_modified_time .
It's fine. It's better to do less hardcode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-728754094
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636233702
cc @cloud-fan @viirya @maropu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dilipbiswal commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
dilipbiswal commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-649105089
@ulysses-you Thanks. Do we know the behavior of hive for this case i.e parquet to orc or vice versa ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636349376
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r432794399
##########
File path: docs/sql-ref-syntax-ddl-create-table-like.md
##########
@@ -22,6 +22,9 @@ license: |
### Description
The `CREATE TABLE` statement defines a new table using the definition/metadata of an existing table or view.
+And we copied from the source table are storage(inputFormat, outputFormat, serde, compressed, properties), schema,
+provider, partitionColumnNames, bucketSpec, tblproperties
+by default.
Review comment:
Plz write this note in the `TABLEPROPERTIES` section instead of doing it here: https://github.com/apache/spark/blame/master/docs/sql-ref-syntax-ddl-create-table-like.md#L59
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636403542
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-720568294
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636296713
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652202119
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-728757909
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635684815
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636349376
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r433130893
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -839,6 +839,19 @@ case class AlterTableSetLocationCommand(
object DDLUtils {
val HIVE_PROVIDER = "hive"
+ val METASTORE_GENERATED_PROPERTIES: Set[String] = Set(
Review comment:
`HiveClientImpl.HiveStatisticsProperties` has been used at `HiveClientImpl.getTable()`, we needn't remove them again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635698603
**[Test build #123256 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123256/testReport)** for PR 28647 at commit [`162627a`](https://github.com/apache/spark/commit/162627ad2f2bc249bfd11580821a0b5de4c5ff6a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635223865
let's update the doc of `CreateTableLikeCommand` to make it clear that we copy table properties as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r432794003
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -2720,4 +2704,26 @@ class HiveDDLSuite
checkAnswer(sql("SHOW PARTITIONS ta_part"), Row("ts=10") :: Nil)
}
}
+
+ test("SPARK-31828: Retain table properties at CreateTableLikeCommand") {
+ val catalog = spark.sessionState.catalog
+ withTable("t1", "t2", "t3") {
+ sql(s"CREATE TABLE t1(c1 int) TBLPROPERTIES('k1'='v1', 'k2'='v2', 'totalNumberFiles'='meta')")
+ val t1 = catalog.getTableMetadata(TableIdentifier("t1"))
+ assert(t1.properties("k1") == "v1")
+ assert(t1.properties("k2") == "v2")
+ assert(t1.properties(s"totalNumberFiles") == "meta")
+ sql("CREATE TABLE t2 LIKE t1 TBLPROPERTIES('k2'='v3', 'k4'='v4')")
+ val t2 = catalog.getTableMetadata(TableIdentifier("t2"))
+ assert(t2.properties("k1") == "v1")
+ assert(t2.properties("k2") == "v3")
+ assert(t2.properties("k4") == "v4")
+ assert(t2.properties.get(s"totalNumberFiles").isEmpty)
+ sql("CREATE TABLE t3 LIKE t1")
+ val t3 = catalog.getTableMetadata(TableIdentifier("t3"))
+ assert(t3.properties("k1") == "v1")
+ assert(t3.properties("k2") == "v2")
+ assert(t3.properties.get(s"totalNumberFiles").isEmpty)
Review comment:
nit: remove `s` in the head.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636013906
**[Test build #123291 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123291/testReport)** for PR 28647 at commit [`ed0877e`](https://github.com/apache/spark/commit/ed0877e10dba240c5a5200888c9f6809b07a4939).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636014527
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650674430
**[Test build #124582 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124582/testReport)** for PR 28647 at commit [`183c209`](https://github.com/apache/spark/commit/183c2090281171519785a16bfa29231dd51c8d8a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-656413247
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r446910478
##########
File path: docs/sql-ref-syntax-ddl-create-table-like.md
##########
@@ -57,6 +57,8 @@ CREATE TABLE [IF NOT EXISTS] table_identifier LIKE source_table_identifier
* **TBLPROPERTIES**
Table properties that have to be set are specified, such as `created.by.user`, `owner`, etc.
+ Note that a basic set of table properties defined in a source table is copied into a new table.
Review comment:
updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-649153781
Hive use it's own way to copy properties, all useful properties which need to copy are declared at serde class annotation so that Hive needn't copy redundant properties.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-648572023
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-655220909
**[Test build #125274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125274/testReport)** for PR 28647 at commit [`19f398b`](https://github.com/apache/spark/commit/19f398b079a087f9c5e93cfc380768f9ae1b6e08).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-655245697
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125274/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635913488
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636349281
**[Test build #123318 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123318/testReport)** for PR 28647 at commit [`bc52948`](https://github.com/apache/spark/commit/bc52948591393402d4d78ba71bb9f58fe81132fc).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635715157
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #28647:
URL: https://github.com/apache/spark/pull/28647
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652467305
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650673555
**[Test build #124581 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124581/testReport)** for PR 28647 at commit [`5c63477`](https://github.com/apache/spark/commit/5c634779f429ae148ff2d7f0453e3935109b7785).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-653164552
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r432941526
##########
File path: docs/sql-ref-syntax-ddl-create-table-like.md
##########
@@ -57,6 +57,7 @@ CREATE TABLE [IF NOT EXISTS] table_identifier LIKE source_table_identifier
* **TBLPROPERTIES**
Table properties that have to be set are specified, such as `created.by.user`, `owner`, etc.
+ And we copied from the source table are storage(inputFormat, outputFormat, serde, compressed, properties), schema, provider, partitionColumnNames, bucketSpec, tblproperties by default.
Review comment:
```
Note that a basic set of table properties defined in a source table is copied into a new table. If a specified property key has already existed in a source table, the old property is overwritten with the new one.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r430943140
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -2632,7 +2632,6 @@ class HiveDDLSuite
assert(source.properties("a") == "apple")
sql("CREATE TABLE t LIKE s STORED AS parquet TBLPROPERTIES('f'='foo', 'b'='bar')")
val table = catalog.getTableMetadata(TableIdentifier("t"))
- assert(table.properties.get("a") === None)
Review comment:
Seems like we intentionally don't keep the table properties from the original table. @maropu @dongjoon-hyun @viirya are you OK with the behavior change proposed by this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-728721434
**[Test build #131201 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131201/testReport)** for PR 28647 at commit [`4b55575`](https://github.com/apache/spark/commit/4b555750488a5c5c77077dbb0aa98514eb04b03f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-723709565
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636321651
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-634373422
One point.
Hive only retain the useful properties through serde class annotation.
Should we remove the properties like `transient_lastDdlTime` simply?
Or follow the hive process ?
cc @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650697842
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-648635481
**[Test build #124466 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124466/testReport)** for PR 28647 at commit [`2251181`](https://github.com/apache/spark/commit/22511814fe2626961ee3eb273ac0a48172faa35c).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r432793971
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -2720,4 +2704,26 @@ class HiveDDLSuite
checkAnswer(sql("SHOW PARTITIONS ta_part"), Row("ts=10") :: Nil)
}
}
+
+ test("SPARK-31828: Retain table properties at CreateTableLikeCommand") {
+ val catalog = spark.sessionState.catalog
+ withTable("t1", "t2", "t3") {
+ sql(s"CREATE TABLE t1(c1 int) TBLPROPERTIES('k1'='v1', 'k2'='v2', 'totalNumberFiles'='meta')")
Review comment:
Just in case, could you test all the prpos in `METASTORE_GENERATED_PROPERTIES`? Probably, its beetter to split it into two test units like add a new test unit `test("SPARK-31828: Filters out Hive metastore properties in CreateTableLikeCommand") {`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r448066210
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##########
@@ -115,6 +116,26 @@ case class CreateTableLikeCommand(
CatalogTableType.EXTERNAL
}
+ // We only copy source tbl properties if the format is the same with each other
+ val needCopyProperties =
+ (provider.isEmpty || provider == sourceTableDesc.provider) &&
Review comment:
ok, sounds good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r433872605
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -839,6 +839,19 @@ case class AlterTableSetLocationCommand(
object DDLUtils {
val HIVE_PROVIDER = "hive"
+ val METASTORE_GENERATED_PROPERTIES: Set[String] = Set(
+ "CreateTime",
+ "transient_lastDdlTime",
+ "grantTime",
+ "lastUpdateTime",
+ "last_modified_by",
+ "last_modified_time",
+ "Owner:",
+ "totalNumberFiles",
+ "maxFileSize",
+ "minFileSize"
Review comment:
OK.
But we can't use hive constants in sql module because of dependency, it seems like hardcode is necessary. Is it fine ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635814973
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635698883
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635746316
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123256/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652174928
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29351/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636144919
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636144919
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r448055387
##########
File path: docs/sql-ref-syntax-ddl-create-table-like.md
##########
@@ -57,6 +57,8 @@ CREATE TABLE [IF NOT EXISTS] table_identifier LIKE source_table_identifier
* **TBLPROPERTIES**
Table properties that have to be set are specified, such as `created.by.user`, `owner`, etc.
+ Note that a basic set of table properties defined in a source table is copied into a new table if the format is the same with each other.
Review comment:
of course
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635715025
**[Test build #123255 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123255/testReport)** for PR 28647 at commit [`b30d62c`](https://github.com/apache/spark/commit/b30d62c884858cdfb065c19ef5dc3115aaa38a29).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652060788
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r448065211
##########
File path: docs/sql-ref-syntax-ddl-create-table-like.md
##########
@@ -57,6 +57,8 @@ CREATE TABLE [IF NOT EXISTS] table_identifier LIKE source_table_identifier
* **TBLPROPERTIES**
Table properties that have to be set are specified, such as `created.by.user`, `owner`, etc.
+ Note that a basic set of table properties defined in a source table is copied into a new table if the table format including data source provider and storage input format are the same.
Review comment:
nit: `the table format including data source provider and storage input format` -> `the table formats including data source providers and storage input formats`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635715157
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-728721434
**[Test build #131201 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131201/testReport)** for PR 28647 at commit [`4b55575`](https://github.com/apache/spark/commit/4b555750488a5c5c77077dbb0aa98514eb04b03f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650708360
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124582/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635704603
**[Test build #123257 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123257/testReport)** for PR 28647 at commit [`9eacf1e`](https://github.com/apache/spark/commit/9eacf1e041dd0ace3efab27777825d20549af0b4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636289558
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r433580610
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -839,6 +839,19 @@ case class AlterTableSetLocationCommand(
object DDLUtils {
val HIVE_PROVIDER = "hive"
+ val METASTORE_GENERATED_PROPERTIES: Set[String] = Set(
+ "CreateTime",
+ "transient_lastDdlTime",
+ "grantTime",
+ "lastUpdateTime",
+ "last_modified_by",
+ "last_modified_time",
+ "Owner:",
+ "totalNumberFiles",
+ "maxFileSize",
+ "minFileSize"
Review comment:
In fact, I can't find `Owner` in tblproperties in hive code. We can try `org.apache.hadoop.hive.metastore.api.hive_metastoreConstants`, but this is also not complete. We need to add some properties like `last_modified_by `, `last_modified_time `.
```
public class hive_metastoreConstants {
public static final String DDL_TIME = "transient_lastDdlTime";
public static final String HIVE_FILTER_FIELD_OWNER = "hive_filter_field_owner__";
public static final String HIVE_FILTER_FIELD_PARAMS = "hive_filter_field_params__";
public static final String HIVE_FILTER_FIELD_LAST_ACCESS = "hive_filter_field_last_access__";
public static final String IS_ARCHIVED = "is_archived";
public static final String ORIGINAL_LOCATION = "original_location";
public static final String IS_IMMUTABLE = "immutable";
public static final String META_TABLE_COLUMNS = "columns";
public static final String META_TABLE_COLUMN_TYPES = "columns.types";
public static final String BUCKET_FIELD_NAME = "bucket_field_name";
public static final String BUCKET_COUNT = "bucket_count";
public static final String FIELD_TO_DIMENSION = "field_to_dimension";
public static final String META_TABLE_NAME = "name";
public static final String META_TABLE_DB = "db";
public static final String META_TABLE_LOCATION = "location";
public static final String META_TABLE_SERDE = "serde";
public static final String META_TABLE_PARTITION_COLUMNS = "partition_columns";
public static final String META_TABLE_PARTITION_COLUMN_TYPES = "partition_columns.types";
public static final String FILE_INPUT_FORMAT = "file.inputformat";
public static final String FILE_OUTPUT_FORMAT = "file.outputformat";
public static final String META_TABLE_STORAGE = "storage_handler";
public static final String TABLE_IS_TRANSACTIONAL = "transactional";
public static final String TABLE_NO_AUTO_COMPACT = "no_auto_compaction";
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650697842
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635749256
**[Test build #123257 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123257/testReport)** for PR 28647 at commit [`9eacf1e`](https://github.com/apache/spark/commit/9eacf1e041dd0ace3efab27777825d20549af0b4).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-634369793
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635686780
**[Test build #123255 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123255/testReport)** for PR 28647 at commit [`b30d62c`](https://github.com/apache/spark/commit/b30d62c884858cdfb065c19ef5dc3115aaa38a29).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-651978119
**[Test build #124686 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124686/testReport)** for PR 28647 at commit [`fcc8b3b`](https://github.com/apache/spark/commit/fcc8b3bcf0141528f57d799c531cc01d8b9aeb4f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652303364
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635704865
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-648618472
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650697664
**[Test build #124581 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124581/testReport)** for PR 28647 at commit [`5c63477`](https://github.com/apache/spark/commit/5c634779f429ae148ff2d7f0453e3935109b7785).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652726488
**[Test build #124836 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124836/testReport)** for PR 28647 at commit [`1e1349f`](https://github.com/apache/spark/commit/1e1349fdff0ee0b36e342a3938734c887d9e93bc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652174626
**[Test build #124739 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124739/testReport)** for PR 28647 at commit [`7f9f685`](https://github.com/apache/spark/commit/7f9f68571a535c2ecb46f9036e4988167416f49f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650697849
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124581/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r447999171
##########
File path: docs/sql-ref-syntax-ddl-create-table-like.md
##########
@@ -57,6 +57,8 @@ CREATE TABLE [IF NOT EXISTS] table_identifier LIKE source_table_identifier
* **TBLPROPERTIES**
Table properties that have to be set are specified, such as `created.by.user`, `owner`, etc.
+ Note that a basic set of table properties defined in a source table is copied into a new table if the format is the same with each other.
Review comment:
Can we be more specified here? Like `... if the table format including data source provider and storage input format are the same...`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635686780
**[Test build #123255 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123255/testReport)** for PR 28647 at commit [`b30d62c`](https://github.com/apache/spark/commit/b30d62c884858cdfb065c19ef5dc3115aaa38a29).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635704603
**[Test build #123257 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123257/testReport)** for PR 28647 at commit [`9eacf1e`](https://github.com/apache/spark/commit/9eacf1e041dd0ace3efab27777825d20549af0b4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636321507
**[Test build #123318 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123318/testReport)** for PR 28647 at commit [`bc52948`](https://github.com/apache/spark/commit/bc52948591393402d4d78ba71bb9f58fe81132fc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-720610136
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652825074
**[Test build #124836 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124836/testReport)** for PR 28647 at commit [`1e1349f`](https://github.com/apache/spark/commit/1e1349fdff0ee0b36e342a3938734c887d9e93bc).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652997899
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650708230
**[Test build #124582 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124582/testReport)** for PR 28647 at commit [`183c209`](https://github.com/apache/spark/commit/183c2090281171519785a16bfa29231dd51c8d8a).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dilipbiswal commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
dilipbiswal commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-651566036
Changes look good to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-648522303
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652060545
**[Test build #124686 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124686/testReport)** for PR 28647 at commit [`fcc8b3b`](https://github.com/apache/spark/commit/fcc8b3bcf0141528f57d799c531cc01d8b9aeb4f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r434403470
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -839,6 +839,19 @@ case class AlterTableSetLocationCommand(
object DDLUtils {
val HIVE_PROVIDER = "hive"
+ val METASTORE_GENERATED_PROPERTIES: Set[String] = Set(
+ "CreateTime",
+ "transient_lastDdlTime",
+ "grantTime",
+ "lastUpdateTime",
+ "last_modified_by",
+ "last_modified_time",
+ "Owner:",
+ "totalNumberFiles",
+ "maxFileSize",
+ "minFileSize"
Review comment:
I think no.
This behavior is only at `CreateTableLikeCommand`. `HiveClientImpl.convertHiveTableToCatalogTable` will affect other opt that use `CatalogTable`, like `desc formatted table`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650925029
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r432208324
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -2720,4 +2708,23 @@ class HiveDDLSuite
checkAnswer(sql("SHOW PARTITIONS ta_part"), Row("ts=10") :: Nil)
}
}
+
+ test("SPARK-31828: Retain table properties at CreateTableLikeCommand") {
+ val catalog = spark.sessionState.catalog
+ withTable("t1", "t2", "t3") {
+ sql("CREATE TABLE t1(c1 int) TBLPROPERTIES('k1'='v1', 'k2'='v2')")
+ val t1 = catalog.getTableMetadata(TableIdentifier("t1"))
+ assert(t1.properties("k1") == "v1")
+ assert(t1.properties("k2") == "v2")
+ sql("CREATE TABLE t2 LIKE t1 TBLPROPERTIES('k2'='v3', 'k4'='v4')")
+ val t2 = catalog.getTableMetadata(TableIdentifier("t2"))
+ assert(t2.properties("k1") == "v1")
+ assert(t2.properties("k2") == "v3")
+ assert(t2.properties("k4") == "v4")
+ sql("CREATE TABLE t3 LIKE t1")
+ val t3 = catalog.getTableMetadata(TableIdentifier("t3"))
+ assert(t3.properties("k1") == "v1")
+ assert(t3.properties("k2") == "v2")
Review comment:
Sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-651978334
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636014527
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652134377
**[Test build #124710 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124710/testReport)** for PR 28647 at commit [`006ec47`](https://github.com/apache/spark/commit/006ec47700154a20ff958feec6cc460327cbbb2f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652060792
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124686/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r446789203
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -839,6 +839,19 @@ case class AlterTableSetLocationCommand(
object DDLUtils {
val HIVE_PROVIDER = "hive"
+ val METASTORE_GENERATED_PROPERTIES: Set[String] = Set(
Review comment:
I don't feel comfortable maintaining this list here. Is it possible to remove Hive specific properties in `HiveClientImpl`? So that the hive related stuff remains in hive related classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r452298973
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -2721,6 +2707,68 @@ class HiveDDLSuite
}
}
+ test("SPARK-31828: Retain table properties at CreateTableLikeCommand") {
+ val catalog = spark.sessionState.catalog
+ withTable("t1", "t2", "t3", "t4", "t5", "t6") {
+ sql("CREATE TABLE t1(c1 int) TBLPROPERTIES('k1'='v1', 'k2'='v2')")
Review comment:
can we add `USING hive`? otherwise if we change the default table format, this test will be broken.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636321507
**[Test build #123318 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123318/testReport)** for PR 28647 at commit [`bc52948`](https://github.com/apache/spark/commit/bc52948591393402d4d78ba71bb9f58fe81132fc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650987295
**[Test build #124635 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124635/testReport)** for PR 28647 at commit [`183c209`](https://github.com/apache/spark/commit/183c2090281171519785a16bfa29231dd51c8d8a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-720570503
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-723772119
cc @maropu @cloud-fan @viirya do you have time to review this thanks !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635684815
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635746311
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652123864
**[Test build #124710 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124710/testReport)** for PR 28647 at commit [`006ec47`](https://github.com/apache/spark/commit/006ec47700154a20ff958feec6cc460327cbbb2f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652995379
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r448002140
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##########
@@ -115,6 +116,26 @@ case class CreateTableLikeCommand(
CatalogTableType.EXTERNAL
}
+ // We only copy source tbl properties if the format is the same with each other
+ val needCopyProperties =
+ (provider.isEmpty || provider == sourceTableDesc.provider) &&
Review comment:
Why not just compare `newProvider` with `sourceTableDesc.provider`?
When `provider.isEmpty` is true, it covers view case, Hive provider, do we need to copy properties for all these cases?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635746311
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636288780
**[Test build #123311 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123311/testReport)** for PR 28647 at commit [`4507e7f`](https://github.com/apache/spark/commit/4507e7f03cea0d080b94c15c0c8db7e601ef3fac).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-723709555
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35369/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r444589954
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -839,6 +839,25 @@ case class AlterTableSetLocationCommand(
object DDLUtils {
val HIVE_PROVIDER = "hive"
+ val RESERVED_PROPERTIES: Set[String] =
+ // spark reserved properties
+ Set(
+ TableCatalog.PROP_COMMENT,
+ TableCatalog.PROP_LOCATION,
+ TableCatalog.PROP_OWNER,
+ TableCatalog.PROP_PROVIDER) ++
Review comment:
Add spark reserved properties.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r432539576
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -1531,21 +1531,6 @@ class HiveDDLSuite
assert(targetTable.unsupportedFeatures.isEmpty,
"the unsupportedFeatures in the create table must be empty")
- val metastoreGeneratedProperties = Seq(
- "CreateTime",
- "transient_lastDdlTime",
- "grantTime",
- "lastUpdateTime",
- "last_modified_by",
- "last_modified_time",
- "Owner:",
- "totalNumberFiles",
- "maxFileSize",
- "minFileSize"
- )
- assert(targetTable.properties.filterKeys(!metastoreGeneratedProperties.contains(_)).isEmpty,
Review comment:
Remove this check. And we can't check all meta properties here. For example `transient_lastDdlTime`, hive will add this properties when table created. So the properties is always exists.
Add some meta properties test at new UT.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652201843
**[Test build #124739 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124739/testReport)** for PR 28647 at commit [`7f9f685`](https://github.com/apache/spark/commit/7f9f68571a535c2ecb46f9036e4988167416f49f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636296713
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r432308450
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##########
@@ -115,6 +116,15 @@ case class CreateTableLikeCommand(
CatalogTableType.EXTERNAL
}
+ val newProperties = sourceTableDesc.tableType match {
+ case VIEW =>
+ // For view, we just use new properties
+ properties
Review comment:
Keep view behavior as before. Hive also does not copy view properties.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r432201354
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##########
@@ -115,6 +116,8 @@ case class CreateTableLikeCommand(
CatalogTableType.EXTERNAL
}
+ val newProperties =
Review comment:
plz leave some comments about why the metastore props are remvoed here? https://github.com/apache/spark/pull/28647#issuecomment-634373422
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-711453288
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635749462
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123257/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-733150165
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636296207
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652202119
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652726840
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-649497380
> cc @cloud-fan @viirya @maropu - Should we be more restrictive here and only copy properties if target and source format and serde match up ? or its okay as is ?
Ah, I see. I think the properties are only meaningful if the format is the same with each other. So, IMHO its okay to copy them only if the format is the same.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-651978119
**[Test build #124686 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124686/testReport)** for PR 28647 at commit [`fcc8b3b`](https://github.com/apache/spark/commit/fcc8b3bcf0141528f57d799c531cc01d8b9aeb4f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636550311
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636289558
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652174925
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635698883
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635814387
**[Test build #123272 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123272/testReport)** for PR 28647 at commit [`175d0e2`](https://github.com/apache/spark/commit/175d0e29fe5c53af0da3d8bbae7156eee1416f69).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635698603
**[Test build #123256 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123256/testReport)** for PR 28647 at commit [`162627a`](https://github.com/apache/spark/commit/162627ad2f2bc249bfd11580821a0b5de4c5ff6a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-648522303
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-791068763
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652174925
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652997001
**[Test build #124906 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124906/testReport)** for PR 28647 at commit [`1e1349f`](https://github.com/apache/spark/commit/1e1349fdff0ee0b36e342a3938734c887d9e93bc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r434357214
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -839,6 +839,19 @@ case class AlterTableSetLocationCommand(
object DDLUtils {
val HIVE_PROVIDER = "hive"
+ val METASTORE_GENERATED_PROPERTIES: Set[String] = Set(
+ "CreateTime",
+ "transient_lastDdlTime",
+ "grantTime",
+ "lastUpdateTime",
+ "last_modified_by",
+ "last_modified_time",
+ "Owner:",
+ "totalNumberFiles",
+ "maxFileSize",
+ "minFileSize"
Review comment:
can we remove these properties in `HiveClientImpl.convertHiveTableToCatalogTable`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-656412962
**[Test build #125525 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125525/testReport)** for PR 28647 at commit [`03966ae`](https://github.com/apache/spark/commit/03966ae6b43d5c4cfbf75ecd492a28cf839287e1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-656446804
**[Test build #125525 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125525/testReport)** for PR 28647 at commit [`03966ae`](https://github.com/apache/spark/commit/03966ae6b43d5c4cfbf75ecd492a28cf839287e1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-728757917
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131201/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652997899
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636421984
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r451227746
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -1230,16 +1230,10 @@ private[hive] object HiveClientImpl extends Logging {
// Visible for testing.
private[hive] val HIVE_METASTORE_GENERATED_PROPERTIES: Set[String] = Set(
Review comment:
After check Hive1.2 and Hive2.3 again, I think we can just remove 3 properties so that we can reduce the scope of influence.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635814973
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r432208610
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -839,6 +839,19 @@ case class AlterTableSetLocationCommand(
object DDLUtils {
val HIVE_PROVIDER = "hive"
+ val METASTORE_GENERATED_PROPERTIES: Set[String] = Set(
Review comment:
I searched somewhere, but I think there is no determinate way to check this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635913194
**[Test build #123272 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123272/testReport)** for PR 28647 at commit [`175d0e2`](https://github.com/apache/spark/commit/175d0e29fe5c53af0da3d8bbae7156eee1416f69).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652826149
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636464402
Looks fine except for one comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r434406585
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -839,6 +839,19 @@ case class AlterTableSetLocationCommand(
object DDLUtils {
val HIVE_PROVIDER = "hive"
+ val METASTORE_GENERATED_PROPERTIES: Set[String] = Set(
+ "CreateTime",
+ "transient_lastDdlTime",
+ "grantTime",
+ "lastUpdateTime",
+ "last_modified_by",
+ "last_modified_time",
+ "Owner:",
+ "totalNumberFiles",
+ "maxFileSize",
+ "minFileSize"
Review comment:
Other hand, it's no matter add some unexpected properties into new table properties.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636288879
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635746196
**[Test build #123256 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123256/testReport)** for PR 28647 at commit [`162627a`](https://github.com/apache/spark/commit/162627ad2f2bc249bfd11580821a0b5de4c5ff6a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652997001
**[Test build #124906 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124906/testReport)** for PR 28647 at commit [`1e1349f`](https://github.com/apache/spark/commit/1e1349fdff0ee0b36e342a3938734c887d9e93bc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-723694297
**[Test build #130760 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130760/testReport)** for PR 28647 at commit [`2d470fc`](https://github.com/apache/spark/commit/2d470fcc236441a1417b073aae7c6c2c697eaae8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-656447058
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125525/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r432202932
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -839,6 +839,19 @@ case class AlterTableSetLocationCommand(
object DDLUtils {
val HIVE_PROVIDER = "hive"
+ val METASTORE_GENERATED_PROPERTIES: Set[String] = Set(
Review comment:
Just to check; does the latest hive metastore have the same set of these props here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635749458
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-723756911
**[Test build #130760 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130760/testReport)** for PR 28647 at commit [`2d470fc`](https://github.com/apache/spark/commit/2d470fcc236441a1417b073aae7c6c2c697eaae8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-655220909
**[Test build #125274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125274/testReport)** for PR 28647 at commit [`19f398b`](https://github.com/apache/spark/commit/19f398b079a087f9c5e93cfc380768f9ae1b6e08).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652136333
> Can you address the comment https://github.com/apache/spark/pull/28647/files#r446789203 too?
Sorry for missed this, I will check it later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-653164552
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636296207
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636295075
@maropu please help test, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r433213091
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -839,6 +839,19 @@ case class AlterTableSetLocationCommand(
object DDLUtils {
val HIVE_PROVIDER = "hive"
+ val METASTORE_GENERATED_PROPERTIES: Set[String] = Set(
+ "CreateTime",
+ "transient_lastDdlTime",
+ "grantTime",
+ "lastUpdateTime",
+ "last_modified_by",
+ "last_modified_time",
+ "Owner:",
+ "totalNumberFiles",
+ "maxFileSize",
+ "minFileSize"
Review comment:
This list was initially created for tests, so we didn't carefully check it. Is it possible to get these property names from some constants in the hive lib? like `HiveClientImpl.HiveStatisticsProperties`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r432794003
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -2720,4 +2704,26 @@ class HiveDDLSuite
checkAnswer(sql("SHOW PARTITIONS ta_part"), Row("ts=10") :: Nil)
}
}
+
+ test("SPARK-31828: Retain table properties at CreateTableLikeCommand") {
+ val catalog = spark.sessionState.catalog
+ withTable("t1", "t2", "t3") {
+ sql(s"CREATE TABLE t1(c1 int) TBLPROPERTIES('k1'='v1', 'k2'='v2', 'totalNumberFiles'='meta')")
+ val t1 = catalog.getTableMetadata(TableIdentifier("t1"))
+ assert(t1.properties("k1") == "v1")
+ assert(t1.properties("k2") == "v2")
+ assert(t1.properties(s"totalNumberFiles") == "meta")
+ sql("CREATE TABLE t2 LIKE t1 TBLPROPERTIES('k2'='v3', 'k4'='v4')")
+ val t2 = catalog.getTableMetadata(TableIdentifier("t2"))
+ assert(t2.properties("k1") == "v1")
+ assert(t2.properties("k2") == "v3")
+ assert(t2.properties("k4") == "v4")
+ assert(t2.properties.get(s"totalNumberFiles").isEmpty)
+ sql("CREATE TABLE t3 LIKE t1")
+ val t3 = catalog.getTableMetadata(TableIdentifier("t3"))
+ assert(t3.properties("k1") == "v1")
+ assert(t3.properties("k2") == "v2")
+ assert(t3.properties.get(s"totalNumberFiles").isEmpty)
Review comment:
nit: remove `s` in the head. It seems there are the same unnecessary `s` in the other places, too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r446788640
##########
File path: docs/sql-ref-syntax-ddl-create-table-like.md
##########
@@ -57,6 +57,8 @@ CREATE TABLE [IF NOT EXISTS] table_identifier LIKE source_table_identifier
* **TBLPROPERTIES**
Table properties that have to be set are specified, such as `created.by.user`, `owner`, etc.
+ Note that a basic set of table properties defined in a source table is copied into a new table.
Review comment:
let's update the doc as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-634369793
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636321651
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-651978334
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635691563
Could you update the SQL doc for the behaivour, too? https://github.com/apache/spark/blame/master/docs/sql-ref-syntax-ddl-create-table-like.md
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r448230855
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##########
@@ -115,6 +116,26 @@ case class CreateTableLikeCommand(
CatalogTableType.EXTERNAL
}
+ // We only copy source tbl properties if the format is the same with each other
+ val needCopyProperties =
+ (provider.isEmpty || provider == sourceTableDesc.provider) &&
Review comment:
added more test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r432203235
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -2720,4 +2708,23 @@ class HiveDDLSuite
checkAnswer(sql("SHOW PARTITIONS ta_part"), Row("ts=10") :: Nil)
}
}
+
+ test("SPARK-31828: Retain table properties at CreateTableLikeCommand") {
+ val catalog = spark.sessionState.catalog
+ withTable("t1", "t2", "t3") {
+ sql("CREATE TABLE t1(c1 int) TBLPROPERTIES('k1'='v1', 'k2'='v2')")
+ val t1 = catalog.getTableMetadata(TableIdentifier("t1"))
+ assert(t1.properties("k1") == "v1")
+ assert(t1.properties("k2") == "v2")
+ sql("CREATE TABLE t2 LIKE t1 TBLPROPERTIES('k2'='v3', 'k4'='v4')")
+ val t2 = catalog.getTableMetadata(TableIdentifier("t2"))
+ assert(t2.properties("k1") == "v1")
+ assert(t2.properties("k2") == "v3")
+ assert(t2.properties("k4") == "v4")
+ sql("CREATE TABLE t3 LIKE t1")
+ val t3 = catalog.getTableMetadata(TableIdentifier("t3"))
+ assert(t3.properties("k1") == "v1")
+ assert(t3.properties("k2") == "v2")
Review comment:
Could you check if the metastore props are filtered out here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636295583
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635814387
**[Test build #123272 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123272/testReport)** for PR 28647 at commit [`175d0e2`](https://github.com/apache/spark/commit/175d0e29fe5c53af0da3d8bbae7156eee1416f69).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636289560
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123311/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r432794744
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##########
@@ -115,6 +116,15 @@ case class CreateTableLikeCommand(
CatalogTableType.EXTERNAL
}
+ val newProperties = sourceTableDesc.tableType match {
+ case VIEW =>
+ // For view, we just use new properties
+ properties
Review comment:
nit format;
```
val newProperties = sourceTableDesc.tableType match {
case MANAGED | EXTERNAL =>
// Hive only retain the useful properties through serde class annotation.
// For better compatible with Hive, we remove the metastore properties.
sourceTableDesc.properties -- DDLUtils.METASTORE_GENERATED_PROPERTIES ++ properties
case VIEW =>
// For view, we just use new properties
properties
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-648617855
Seems we only support the v1 catalog for `CreateTableLike`, don't need check reserved properties in `TableCatalog`.
BTW there is a work with `CreateTableLike` for v2 catalog. [SPARK-29563](https://github.com/apache/spark/pull/26219)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636289548
**[Test build #123311 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123311/testReport)** for PR 28647 at commit [`4507e7f`](https://github.com/apache/spark/commit/4507e7f03cea0d080b94c15c0c8db7e601ef3fac).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-653163277
**[Test build #124906 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124906/testReport)** for PR 28647 at commit [`1e1349f`](https://github.com/apache/spark/commit/1e1349fdff0ee0b36e342a3938734c887d9e93bc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-648635631
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124466/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-648572023
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r448065805
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##########
@@ -55,7 +55,8 @@ import org.apache.spark.sql.util.SchemaUtils
* are identical to the ones defined in the source table.
*
* The CatalogTable attributes copied from the source table are storage(inputFormat, outputFormat,
- * serde, compressed, properties), schema, provider, partitionColumnNames, bucketSpec by default.
+ * serde, compressed, properties), schema, provider, partitionColumnNames, bucketSpec, tblproperties
+ * by default.
*
Review comment:
Plz update the comment, too? `if the formats is the same..`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-656412962
**[Test build #125525 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125525/testReport)** for PR 28647 at commit [`03966ae`](https://github.com/apache/spark/commit/03966ae6b43d5c4cfbf75ecd492a28cf839287e1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-723757613
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650981340
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636550166
**[Test build #123355 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123355/testReport)** for PR 28647 at commit [`46d7a7b`](https://github.com/apache/spark/commit/46d7a7bed663c80b155c6abdd4535ef4f3d58386).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r448066069
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##########
@@ -115,6 +116,26 @@ case class CreateTableLikeCommand(
CatalogTableType.EXTERNAL
}
+ // We only copy source tbl properties if the format is the same with each other
+ val needCopyProperties =
+ (provider.isEmpty || provider == sourceTableDesc.provider) &&
Review comment:
We already have tests for all the case described above?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r448067425
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##########
@@ -115,6 +116,26 @@ case class CreateTableLikeCommand(
CatalogTableType.EXTERNAL
}
+ // We only copy source tbl properties if the format is the same with each other
+ val needCopyProperties =
+ (provider.isEmpty || provider == sourceTableDesc.provider) &&
Review comment:
+1 for @maropu's comment. If we have tests for them, it should be good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-732999100
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r452300586
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -1230,16 +1230,10 @@ private[hive] object HiveClientImpl extends Logging {
// Visible for testing.
private[hive] val HIVE_METASTORE_GENERATED_PROPERTIES: Set[String] = Set(
Review comment:
Does this include all the properties in https://github.com/apache/spark/pull/28647/files#diff-b7094baa12601424a5d19cb930e3402fL1534 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650708358
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650674430
**[Test build #124582 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124582/testReport)** for PR 28647 at commit [`183c209`](https://github.com/apache/spark/commit/183c2090281171519785a16bfa29231dd51c8d8a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650673769
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650674535
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636550311
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-728757909
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-733150165
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650673769
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-720610136
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652302561
**[Test build #124764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124764/testReport)** for PR 28647 at commit [`0a3b1cf`](https://github.com/apache/spark/commit/0a3b1cf2ac5bcf0ef77201973ba45ed8f06e731f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635749458
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636296716
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123314/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636403542
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636550166
**[Test build #123355 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123355/testReport)** for PR 28647 at commit [`46d7a7b`](https://github.com/apache/spark/commit/46d7a7bed663c80b155c6abdd4535ef4f3d58386).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636403445
**[Test build #123323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123323/testReport)** for PR 28647 at commit [`6b523e1`](https://github.com/apache/spark/commit/6b523e142267ab5b28072dcfbd493714a75b45b0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636608930
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-648738931
It's a trade-off. In normal copy the redundant properties is little harmful to new table, but not do copy could make a bug. In parquet table case, a source table is compressed with table properties `parquet.compression` then new table should also have the properties. Otherwise the new table is not compressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652726488
**[Test build #124836 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124836/testReport)** for PR 28647 at commit [`1e1349f`](https://github.com/apache/spark/commit/1e1349fdff0ee0b36e342a3938734c887d9e93bc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-655221228
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650673555
**[Test build #124581 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124581/testReport)** for PR 28647 at commit [`5c63477`](https://github.com/apache/spark/commit/5c634779f429ae148ff2d7f0453e3935109b7785).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-648572028
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124441/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635715162
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123255/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r431428218
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
##########
@@ -2632,7 +2632,6 @@ class HiveDDLSuite
assert(source.properties("a") == "apple")
sql("CREATE TABLE t LIKE s STORED AS parquet TBLPROPERTIES('f'='foo', 'b'='bar')")
val table = catalog.getTableMetadata(TableIdentifier("t"))
- assert(table.properties.get("a") === None)
Review comment:
Based on the description of `CreateTableLikeCommand`:
> The CatalogTable attributes copied from the source table are storage(inputFormat, outputFormat, serde, compressed, properties), schema, provider, partitionColumnNames, bucketSpec by default.
So we don't say table properties are copied from source table too. Not sure about why we didn't copy table properties.
I feel it is OK as seems copying original table properties should not be harmful. And we already copy storage properties.
But we should update the doc together.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635704865
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635099609
In `HiveDDLSuite`, exists the [code](https://github.com/apache/spark/blob/d2bec5e265e0aa4fa527c3f43cfe738cdbdc4598/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala#L1546)
```
assert(targetTable.properties.filterKeys(!metastoreGeneratedProperties.contains(_)).isEmpty,
"the table properties of source tables should not be copied in the created table")
```
If use this behavior, we should change the test code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-728754079
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35804/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r448102285
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -1066,7 +1066,10 @@ private[hive] object HiveClientImpl extends Logging {
hiveTable.setSerializationLib(
table.storage.serde.getOrElse("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"))
table.storage.properties.foreach { case (k, v) => hiveTable.setSerdeParam(k, v) }
- table.properties.foreach { case (k, v) => hiveTable.setProperty(k, v) }
+ // Hive only retain the useful properties through serde class annotation.
+ // For better compatible with Hive, we remove the metastore properties.
+ val hiveProperties = table.properties -- HIVE_METASTORE_GENERATED_PROPERTIES
Review comment:
It also affect `createTable`, but seems fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636421984
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636608930
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636296098
**[Test build #123314 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123314/testReport)** for PR 28647 at commit [`4507e7f`](https://github.com/apache/spark/commit/4507e7f03cea0d080b94c15c0c8db7e601ef3fac).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652134496
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124710/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dilipbiswal commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
dilipbiswal commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-651976770
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r448064686
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##########
@@ -115,6 +116,26 @@ case class CreateTableLikeCommand(
CatalogTableType.EXTERNAL
}
+ // We only copy source tbl properties if the format is the same with each other
+ val needCopyProperties =
+ (provider.isEmpty || provider == sourceTableDesc.provider) &&
Review comment:
`inputFormat` only used in Hive provider right ? it will not happen.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-655244977
**[Test build #125274 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125274/testReport)** for PR 28647 at commit [`19f398b`](https://github.com/apache/spark/commit/19f398b079a087f9c5e93cfc380768f9ae1b6e08).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-648635620
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650981328
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-723694297
**[Test build #130760 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130760/testReport)** for PR 28647 at commit [`2d470fc`](https://github.com/apache/spark/commit/2d470fcc236441a1417b073aae7c6c2c697eaae8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r432941526
##########
File path: docs/sql-ref-syntax-ddl-create-table-like.md
##########
@@ -57,6 +57,7 @@ CREATE TABLE [IF NOT EXISTS] table_identifier LIKE source_table_identifier
* **TBLPROPERTIES**
Table properties that have to be set are specified, such as `created.by.user`, `owner`, etc.
+ And we copied from the source table are storage(inputFormat, outputFormat, serde, compressed, properties), schema, provider, partitionColumnNames, bucketSpec, tblproperties by default.
Review comment:
```
Note that a basic set of table properties defined in a source table is copied into a new table.
If a specified property key has already existed in a source table, the old property is overwritten with the new one.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #28647:
URL: https://github.com/apache/spark/pull/28647#discussion_r448066184
##########
File path: docs/sql-ref-syntax-ddl-create-table-like.md
##########
@@ -57,6 +57,8 @@ CREATE TABLE [IF NOT EXISTS] table_identifier LIKE source_table_identifier
* **TBLPROPERTIES**
Table properties that have to be set are specified, such as `created.by.user`, `owner`, etc.
+ Note that a basic set of table properties defined in a source table is copied into a new table if the table format including data source provider and storage input format are the same.
Review comment:
get. thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652174626
**[Test build #124739 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124739/testReport)** for PR 28647 at commit [`7f9f685`](https://github.com/apache/spark/commit/7f9f68571a535c2ecb46f9036e4988167416f49f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652134491
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636013906
**[Test build #123291 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123291/testReport)** for PR 28647 at commit [`ed0877e`](https://github.com/apache/spark/commit/ed0877e10dba240c5a5200888c9f6809b07a4939).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636296711
**[Test build #123314 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123314/testReport)** for PR 28647 at commit [`4507e7f`](https://github.com/apache/spark/commit/4507e7f03cea0d080b94c15c0c8db7e601ef3fac).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dilipbiswal commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
dilipbiswal commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-649476962
Thanks @ulysses-you .
cc @cloud-fan @viirya @maropu - Should we be more restrictive here and only copy properties if target and source format and serde match up ? or its okay as is ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652726840
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652467312
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-648521944
**[Test build #124441 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124441/testReport)** for PR 28647 at commit [`78ff34f`](https://github.com/apache/spark/commit/78ff34f57a8756717a30bbe279fecfee26abe296).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-656413247
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650987295
**[Test build #124635 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124635/testReport)** for PR 28647 at commit [`183c209`](https://github.com/apache/spark/commit/183c2090281171519785a16bfa29231dd51c8d8a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635913488
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dilipbiswal commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
dilipbiswal commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-648661039
@ulysses-you I have a question. Per the syntax the source file format, row format can be different than the target. Say source format was parquet and target is orc, can we copy the parquet specific table properties to the target table ? Like in the pr description we have specified "parquet.compression". Could this be the reason why we didn't copy these properties before ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-732999100
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-656447055
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650674535
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652124188
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-723704368
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35369/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-720570503
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-636349378
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123318/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-648571620
**[Test build #124441 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124441/testReport)** for PR 28647 at commit [`78ff34f`](https://github.com/apache/spark/commit/78ff34f57a8756717a30bbe279fecfee26abe296).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-732791895
**[Test build #131644 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131644/testReport)** for PR 28647 at commit [`c45489a`](https://github.com/apache/spark/commit/c45489ad5b8ddd53d5e81fbba4cd08c0b4fd9850).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-650708358
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-635913493
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123272/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-728754094
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28647:
URL: https://github.com/apache/spark/pull/28647#issuecomment-652826161
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124836/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org