You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/16 08:18:24 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
HyukjinKwon opened a new pull request #30799:
URL: https://github.com/apache/spark/pull/30799
### What changes were proposed in this pull request?
This PR proposes to sort table properties in DESCRIBE TABLE command. This is consistent with DSv2 command as well:
https://github.com/apache/spark/blob/e3058ba17cb4512537953eb4ded884e24ee93ba2/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTableExec.scala#L63
This PR fixes the test case in Scala 2.13 build as well where the table properties have different order in the map.
### Why are the changes needed?
To keep the deterministic and pretty output, and fix the tests in Scala 2.13 build.
See https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-3.2-scala-2.13/49/testReport/junit/org.apache.spark.sql/SQLQueryTestSuite/describe_sql/
```
describe.sql
Expected "...spark_catalog, view.[query.out.col.2=c, view.referredTempFunctionsNames=[], view.catalogAndNamespace.part.1=default]]", but got "...spark_catalog, view.[catalogAndNamespace.part.1=default, view.query.out.col.2=c, view.referredTempFunctionsNames=[]]]" Result did not match for query #29
DESC FORMATTED v
```
### Does this PR introduce _any_ user-facing change?
Yes, it will change the text output from `DESCRIBE [EXTENDED|FORMATTED] table_name`.
Now the table properties are sorted by its key.
### How was this patch tested?
Related unittests were fixed accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30799:
URL: https://github.com/apache/spark/pull/30799#issuecomment-746027031
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37475/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30799:
URL: https://github.com/apache/spark/pull/30799#issuecomment-745974752
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37475/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30799:
URL: https://github.com/apache/spark/pull/30799#issuecomment-746261419
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132873/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30799:
URL: https://github.com/apache/spark/pull/30799#issuecomment-745897954
**[Test build #132873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132873/testReport)** for PR 30799 at commit [`2e908de`](https://github.com/apache/spark/commit/2e908def417f933a993972110df86a43ec0b2f73).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30799:
URL: https://github.com/apache/spark/pull/30799#issuecomment-745897954
**[Test build #132873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132873/testReport)** for PR 30799 at commit [`2e908de`](https://github.com/apache/spark/commit/2e908def417f933a993972110df86a43ec0b2f73).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30799:
URL: https://github.com/apache/spark/pull/30799#issuecomment-746027031
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37475/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30799:
URL: https://github.com/apache/spark/pull/30799#discussion_r544854547
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
##########
@@ -388,7 +388,8 @@ case class CatalogTable(
def toLinkedHashMap: mutable.LinkedHashMap[String, String] = {
val map = new mutable.LinkedHashMap[String, String]()
- val tableProperties = properties.map(p => p._1 + "=" + p._2).mkString("[", ", ", "]")
+ val tableProperties = properties.toSeq.sortBy(_._1)
+ .map(p => p._1 + "=" + p._2).mkString("[", ", ", "]")
Review comment:
ah, sorry, we can guarantee uniqueness of keys.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30799:
URL: https://github.com/apache/spark/pull/30799#issuecomment-745899850
Thanks @dongjoon-hyun! It's super late in your time .. :-).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30799:
URL: https://github.com/apache/spark/pull/30799#issuecomment-746014944
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37475/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30799:
URL: https://github.com/apache/spark/pull/30799#discussion_r544852303
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
##########
@@ -388,7 +388,8 @@ case class CatalogTable(
def toLinkedHashMap: mutable.LinkedHashMap[String, String] = {
val map = new mutable.LinkedHashMap[String, String]()
- val tableProperties = properties.map(p => p._1 + "=" + p._2).mkString("[", ", ", "]")
+ val tableProperties = properties.toSeq.sortBy(_._1)
+ .map(p => p._1 + "=" + p._2).mkString("[", ", ", "]")
Review comment:
Should we sort by entire strings (._1 and ._2)? For example, if you have `key0 = b` and `key0 = a`, you may get unstable output either `[key0 = b, key0 = a]` or `[key0 = a, key0 = b]`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30799:
URL: https://github.com/apache/spark/pull/30799#issuecomment-746301193
thanks, merging to master/3.1!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30799:
URL: https://github.com/apache/spark/pull/30799#issuecomment-745894167
@cloud-fan can you take a quick look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30799:
URL: https://github.com/apache/spark/pull/30799#issuecomment-745903326
It's a catch-up before going to bed. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30799:
URL: https://github.com/apache/spark/pull/30799#issuecomment-746261419
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132873/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #30799:
URL: https://github.com/apache/spark/pull/30799
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30799:
URL: https://github.com/apache/spark/pull/30799#discussion_r544852303
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
##########
@@ -388,7 +388,8 @@ case class CatalogTable(
def toLinkedHashMap: mutable.LinkedHashMap[String, String] = {
val map = new mutable.LinkedHashMap[String, String]()
- val tableProperties = properties.map(p => p._1 + "=" + p._2).mkString("[", ", ", "]")
+ val tableProperties = properties.toSeq.sortBy(_._1)
+ .map(p => p._1 + "=" + p._2).mkString("[", ", ", "]")
Review comment:
Should we sort by entire strings? For example, if you have `key0 = b` and `key0 = a`, you may get unstable output either `[key0 = b, key0 = a]` or `[key0 = a, key0 = b]`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30799: [SPARK-33803][SQL] Sort table properties by key in DESCRIBE TABLE command
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30799:
URL: https://github.com/apache/spark/pull/30799#issuecomment-746249217
**[Test build #132873 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132873/testReport)** for PR 30799 at commit [`2e908de`](https://github.com/apache/spark/commit/2e908def417f933a993972110df86a43ec0b2f73).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org