You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/05/02 01:29:44 UTC

[GitHub] [spark] brkyvz opened a new pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

brkyvz opened a new pull request #28434:
URL: https://github.com/apache/spark/pull/28434


   ## What changes were proposed in this pull request?
   
   SHOW TBLPROPERTIES does not get the correct table properties for tables using the Session Catalog. This PR fixes that, by explicitly falling back to the V1 implementation if the table is in fact a V1 table.
   
   ### Why are the changes needed?
   
   Shows the incorrect table properties, i.e. only what exists in the Hive MetaStore for V2 tables that may have table properties outside of the MetaStore.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Fixes a bug
   
   ### How was this patch tested?
   
   Regression test


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-623182874


   **[Test build #122236 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122236/testReport)** for PR 28434 at commit [`9828286`](https://github.com/apache/spark/commit/98282869d92e609ba6d74a8fa98dcbd0e51080b7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-622670831






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-622649815






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-623182874


   **[Test build #122236 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122236/testReport)** for PR 28434 at commit [`9828286`](https://github.com/apache/spark/commit/98282869d92e609ba6d74a8fa98dcbd0e51080b7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-623220796






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-623182098






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-623220796






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] brkyvz commented on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
brkyvz commented on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-622649369


   It would be great to merge this into Spark 3.0. cc @cloud-fan 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-623220371


   **[Test build #122236 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122236/testReport)** for PR 28434 at commit [`9828286`](https://github.com/apache/spark/commit/98282869d92e609ba6d74a8fa98dcbd0e51080b7).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk commented on a change in pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #28434:
URL: https://github.com/apache/spark/pull/28434#discussion_r418935761



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSessionCatalogSuite.scala
##########
@@ -63,4 +63,27 @@ class DataSourceV2SQLSessionCatalogSuite
       }
     }
   }
+
+  test("SPARK-31624: SHOW TBLPROPERTIES working with V2 tables and the session catalog") {
+    val t1 = "tbl"
+    withTable(t1) {
+      sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format TBLPROPERTIES " +
+        s"(key='v', key2='v2')")

Review comment:
       's' is not needed

##########
File path: sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSessionCatalogSuite.scala
##########
@@ -63,4 +63,27 @@ class DataSourceV2SQLSessionCatalogSuite
       }
     }
   }
+
+  test("SPARK-31624: SHOW TBLPROPERTIES working with V2 tables and the session catalog") {
+    val t1 = "tbl"
+    withTable(t1) {
+      sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format TBLPROPERTIES " +
+        s"(key='v', key2='v2')")
+
+      checkAnswer(
+        sql(s"SHOW TBLPROPERTIES $t1"),
+        Seq(Row("key", "v"), Row("key2", "v2"))
+      )

Review comment:
       Put ')' at the end of previous line

##########
File path: sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSessionCatalogSuite.scala
##########
@@ -63,4 +63,27 @@ class DataSourceV2SQLSessionCatalogSuite
       }
     }
   }
+
+  test("SPARK-31624: SHOW TBLPROPERTIES working with V2 tables and the session catalog") {
+    val t1 = "tbl"
+    withTable(t1) {
+      sql(s"CREATE TABLE $t1 (id bigint, data string) USING $v2Format TBLPROPERTIES " +
+        s"(key='v', key2='v2')")
+
+      checkAnswer(
+        sql(s"SHOW TBLPROPERTIES $t1"),
+        Seq(Row("key", "v"), Row("key2", "v2"))
+      )
+
+      checkAnswer(

Review comment:
       This can fit to 100 chars?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-623431537


   thanks, merging to master/3.0!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-622670731


   **[Test build #122185 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122185/testReport)** for PR 28434 at commit [`d32a6c4`](https://github.com/apache/spark/commit/d32a6c4fd03a6aa1f7e27f7dd9f60f7da2b4583a).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-622649815






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-622670831


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-623182098






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-622670833


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122185/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-622649684


   **[Test build #122185 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122185/testReport)** for PR 28434 at commit [`d32a6c4`](https://github.com/apache/spark/commit/d32a6c4fd03a6aa1f7e27f7dd9f60f7da2b4583a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #28434: [SPARK-31624] Fix SHOW TBLPROPERTIES for V2 tables that leverage the session catalog

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28434:
URL: https://github.com/apache/spark/pull/28434#issuecomment-622649684


   **[Test build #122185 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122185/testReport)** for PR 28434 at commit [`d32a6c4`](https://github.com/apache/spark/commit/d32a6c4fd03a6aa1f7e27f7dd9f60f7da2b4583a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org