You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/20 04:02:00 UTC
[GitHub] [spark] MrPowers opened a new pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
MrPowers opened a new pull request #31254:
URL: https://github.com/apache/spark/pull/31254
### What changes were proposed in this pull request?
Add `count_distinct` was added as an option argument to Dataset#summary (the method already supports count, min, max, etc.)
### Why are the changes needed?
The `summary()` method is used for lightweight exploratory data analysis. A distinct count of all the columns is one of the most common exploratory data analysis queries.
Distinct counts can be expensive, so this shouldn't be enabled by default. The proposed implementation is completely backwards compatible.
### Does this PR introduce _any_ user-facing change?
Yes, users can now call `df.summary("count_distinct")`, which wasn't an option before. Users can still call `df.summary()` without any arguments and the output is the same. `count_distinct` was not added as one of the `defaultStatistics`.
### How was this patch tested?
Unit tests.
### Additional comments
If this idea is accepted, we should add a PySpark implementation in this PR, as suggested by @zero323.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-768405843
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39144/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763865207
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134289/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-768466190
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39144/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763836516
**[Test build #134289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134289/testReport)** for PR 31254 at commit [`bc547a2`](https://github.com/apache/spark/commit/bc547a2c949dba8322378a334d8f7221e46e3784).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-768533838
**[Test build #134558 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134558/testReport)** for PR 31254 at commit [`bc547a2`](https://github.com/apache/spark/commit/bc547a2c949dba8322378a334d8f7221e46e3784).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MrPowers commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
MrPowers commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r561164404
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/stat/StatFunctions.scala
##########
@@ -254,6 +255,8 @@ object StatFunctions extends Logging {
} else {
stats.toLowerCase(Locale.ROOT) match {
case "count" => (child: Expression) => Count(child).toAggregateExpression()
+ case "count_distinct" => (child: Expression) =>
+ UnresolvedFunction("count", Seq(child), isDistinct = true)
Review comment:
Updated, good catch ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r561461000
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
I am not sure. You can all have what you want via `agg`. This is just a shortcut to show basic stats, which was inspired by pandas' one, see https://github.com/apache/spark/commit/e1a172c201d68406faa53b113518b10c879f1ff6. We should of course avoid showing every stats here.
You can easily do `df.agg("col" -> "approx_count_distinct", ...)` easily for other stats. Adding `count_distinct` here makes it ambiguous what to add in `DataFrame.describe` and `DataFrame.summary`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r565353751
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
Sounds good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
srowen closed pull request #31254:
URL: https://github.com/apache/spark/pull/31254
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763859018
**[Test build #134289 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134289/testReport)** for PR 31254 at commit [`bc547a2`](https://github.com/apache/spark/commit/bc547a2c949dba8322378a334d8f7221e46e3784).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763385205
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MrPowers commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
MrPowers commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r561165605
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
##########
@@ -830,6 +830,15 @@ class DataFrameSuite extends QueryTest
("David", 60, 192),
("Amy", 24, 180)).toDF("name", "age", "height")
+ private lazy val person3: DataFrame = Seq(
Review comment:
All the other Dataset#summary related tests are here, so might be best to keep them all in one place. It is a little tricky to find the tests sometimes, so might be good to address the test organization as part of a larger refactoring.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-768337997
Jenkins retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763865207
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134289/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-769111532
Merged to master
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r561465068
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
It's true, though, users looking for summary stats are probably looking at summary(), and sometimes looking for distinct values, and this puts them into one output with the others. I think it's slightly worth adding one more somewhat-common summary stat.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
Sure, but this is only computed if the caller requests it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763357130
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38837/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763898649
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38875/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r561465068
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
It's true, though, users looking for summary stats are probably looking at summary(), and sometimes looking for distinct values, and this puts them into one output with the others. I think it's slightly worth adding one more somewhat-common summary stat.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r561037218
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
Wouldn't be more useful to expose approx variants here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r561911611
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
Sure, but this is only computed if the caller requests it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-768557738
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134558/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r561891981
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
> I am not sure. You can all have what you want via `agg`
Which is also more suitable for anything but quick visual inspection.
My biggest concern is somewhat different though ‒ existing summaries have good performance and predictable behavior. In contrast count distinct can be unstable and approx count distinct slower and more resource hungry.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763314026
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r565016159
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
I'm okay given that we don't change the default.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r560674707
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/stat/StatFunctions.scala
##########
@@ -254,6 +255,8 @@ object StatFunctions extends Logging {
} else {
stats.toLowerCase(Locale.ROOT) match {
case "count" => (child: Expression) => Count(child).toAggregateExpression()
+ case "count_distinct" => (child: Expression) =>
+ UnresolvedFunction("count", Seq(child), isDistinct = true)
Review comment:
`Count(child).toAggregateExpression(isDistinct = true)`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-768485364
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39144/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-768557738
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134558/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763336647
**[Test build #134251 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134251/testReport)** for PR 31254 at commit [`5dc363b`](https://github.com/apache/spark/commit/5dc363bbec02141dd0d58665622b26817d1e8803).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-768367512
**[Test build #134558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134558/testReport)** for PR 31254 at commit [`bc547a2`](https://github.com/apache/spark/commit/bc547a2c949dba8322378a334d8f7221e46e3784).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r560679074
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
I wouldn't add this if there's any strong reason for it because you're already able to do this via `agg`.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
I wouldn't add this if there isn't any strong reason for it because you're already able to do this via `agg`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763336647
**[Test build #134251 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134251/testReport)** for PR 31254 at commit [`5dc363b`](https://github.com/apache/spark/commit/5dc363bbec02141dd0d58665622b26817d1e8803).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MrPowers commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
MrPowers commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r561210774
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
@zero323 - I updated the PR to expose `approx_count_distinct` as well, good point.
[One SO question on this](https://stackoverflow.com/questions/37949494/how-to-count-occurrences-of-each-distinct-value-for-every-column-in-a-dataframe) has 81,000 views and [the other](https://stackoverflow.com/questions/40888946/spark-dataframe-count-distinct-values-of-every-column) has 66,000 views. There is a high demand for count distinct / approx_count_distinct exploratory data analysis queries by columns among the Spark userbase.
These options don't cost us much and should help the less technical Spark users that aren't comfortable using `agg`. I help lots of folks that can't even program and they aren't code geniuses like you all ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r561891981
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
> I am not sure. You can all have what you want via `agg`
Which is also more suitable for anything but quick visual inspection.
My biggest concern is somewhat different though ‒ existing summaries have good performance and predictable behavior. In contrast count distinct can be unstable and approx count distinct slower and more resource hungry.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r560678865
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
I am not sure how common it is to show distinct count. I believe this method was inspired by [`pandas.DataFrame.describe()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.describe.html) which however does not include distinct count.
BTW, I think you should update the examples below too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763836516
**[Test build #134289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134289/testReport)** for PR 31254 at commit [`bc547a2`](https://github.com/apache/spark/commit/bc547a2c949dba8322378a334d8f7221e46e3784).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763855220
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38875/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-768367512
**[Test build #134558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134558/testReport)** for PR 31254 at commit [`bc547a2`](https://github.com/apache/spark/commit/bc547a2c949dba8322378a334d8f7221e46e3784).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-768485364
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39144/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763898649
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38875/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763871419
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38875/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r560886563
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
Meh, I think this could be useful as an _option_; it doesn't really expand the API surface or complexity. I'm OK with adding it, FWIW.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763898649
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38875/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763385205
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763314026
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763374525
**[Test build #134251 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134251/testReport)** for PR 31254 at commit [`5dc363b`](https://github.com/apache/spark/commit/5dc363bbec02141dd0d58665622b26817d1e8803).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r560674949
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
##########
@@ -830,6 +830,15 @@ class DataFrameSuite extends QueryTest
("David", 60, 192),
("Amy", 24, 180)).toDF("name", "age", "height")
+ private lazy val person3: DataFrame = Seq(
Review comment:
Could you move this test into `DataFrameStatSuite`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r561461000
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
I am not sure. You can all have what you want via `agg`. This is just a shortcut to show basic stats, which was inspired by pandas' one, see https://github.com/apache/spark/commit/e1a172c201d68406faa53b113518b10c879f1ff6. We should of course avoid showing every stats here.
You can easily do `df.agg("col" -> "approx_count_distinct", ...)` easily for other stats. Adding `count_distinct` here makes it ambiguous what to add in `DataFrame.describe` and `DataFrame.summary`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763365834
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38837/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763898649
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38875/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #31254:
URL: https://github.com/apache/spark/pull/31254#issuecomment-763327672
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r564935284
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
I would favor this change. If there aren't significant objections, I'd like to merge it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MrPowers commented on a change in pull request #31254: [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary
Posted by GitBox <gi...@apache.org>.
MrPowers commented on a change in pull request #31254:
URL: https://github.com/apache/spark/pull/31254#discussion_r564212822
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2670,6 +2670,7 @@ class Dataset[T] private[sql](
* <li>min</li>
* <li>max</li>
* <li>arbitrary approximate percentiles specified as a percentage (e.g. 75%)</li>
+ * <li>count_distinct</li>
Review comment:
@HyukjinKwon - you raise a good point about API ambiguity between `Dataset.describe` and `Dataset.summary`, but those ambiguities already exist. `df.describe("some_col")`, `df.select("some_col").summary()`, and `df.select("some_col").summary("37%", "87%")` all return different results.
`describe` is a "partial alias" of `summary`, but the method signatures are different.
The Spark & Pandas `describe` methods aren't consistent either. The Spark `describe` method takes column name arguments whereas the Pandas `describe` method takes options, see the Pandas method signature: `DataFrame.describe(percentiles=None, include=None, exclude=None, datetime_is_numeric=False)`.
A Pandas user is already going to be confused by the API. They're going to run `df.describe(percentiles=Array(.25, .5, .75))` and have to Google around to find that they need to run `df.summary("0.25%", "0.5%", "0.75%")`.
Adding additional options to `Dataset.summary` seems consistent with the existing design. The `summary` method is already where we put the extra metrics that aren't enabled by default, like the option to add arbitrary percentiles.
BTW - I don't mean to complain here ;). It's easy for me to make comments on API decisions made years ago from folks in the trenches. You are all doing a wonderful job maintaining this project and continuously moving it in a good direction. I'm always impressed by the code you write and will hopefully get to your level some day. My PR comments sound critical cause I'm just trying to get to the point. I'm very grateful for all the work you've done on this project.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org