You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/12/05 09:18:53 UTC
[GitHub] [spark] zhengruifeng opened a new pull request, #38914: [SPARK-41381][CONNECT][PYTHON] Implement `count_distinct` and `sum_distinct` functions
zhengruifeng opened a new pull request, #38914:
URL: https://github.com/apache/spark/pull/38914
### What changes were proposed in this pull request?
Implement `count_distinct` and `sum_distinct` functions
### Why are the changes needed?
for API coverage
### Does this PR introduce _any_ user-facing change?
new APIs
### How was this patch tested?
added UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] amaliujia commented on a diff in pull request #38914: [SPARK-41381][CONNECT][PYTHON] Implement `count_distinct` and `sum_distinct` functions
Posted by GitBox <gi...@apache.org>.
amaliujia commented on code in PR #38914:
URL: https://github.com/apache/spark/pull/38914#discussion_r1039886159
##########
connector/connect/src/main/protobuf/spark/connect/expressions.proto:
##########
@@ -141,11 +141,14 @@ message Expression {
// (Optional) Function arguments. Empty arguments are allowed.
repeated Expression arguments = 2;
+ // (Required) Indicate if this function should be applied on distinct values.
+ bool is_distinct = 3;
Review Comment:
Do we need a negative test case here where for example if I call `substr` but with this `is_distinct=true`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] amaliujia commented on a diff in pull request #38914: [SPARK-41381][CONNECT][PYTHON] Implement `count_distinct` and `sum_distinct` functions
Posted by GitBox <gi...@apache.org>.
amaliujia commented on code in PR #38914:
URL: https://github.com/apache/spark/pull/38914#discussion_r1039886159
##########
connector/connect/src/main/protobuf/spark/connect/expressions.proto:
##########
@@ -141,11 +141,14 @@ message Expression {
// (Optional) Function arguments. Empty arguments are allowed.
repeated Expression arguments = 2;
+ // (Required) Indicate if this function should be applied on distinct values.
+ bool is_distinct = 3;
Review Comment:
Do we need a negative test case here? For example if I call `substr` but with this `is_distinct=true`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #38914: [SPARK-41381][CONNECT][PYTHON] Implement `count_distinct` and `sum_distinct` functions
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #38914:
URL: https://github.com/apache/spark/pull/38914#issuecomment-1340493981
also cc @cloud-fan @grundprinzip
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #38914: [SPARK-41381][CONNECT][PYTHON] Implement `count_distinct` and `sum_distinct` functions
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #38914:
URL: https://github.com/apache/spark/pull/38914#discussion_r1041728393
##########
python/pyspark/sql/tests/connect/test_connect_function.py:
##########
@@ -413,6 +412,22 @@ def test_aggregation_functions(self):
sdf.groupBy("a").agg(SF.percentile_approx(sdf.b, [0.1, 0.9])).toPandas(),
)
+ # test count_distinct
+ self.assert_eq(
+ cdf.select(CF.count_distinct("b"), CF.count_distinct(cdf.c)).toPandas(),
+ sdf.select(SF.count_distinct("b"), SF.count_distinct(sdf.c)).toPandas(),
+ )
+ # The output column names of 'groupBy.agg(count_distinct)' in PySpark
Review Comment:
this incorrect column name seems not easy to fix, since it should not be related to this PR, I change the column names and compare the values here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #38914: [SPARK-41381][CONNECT][PYTHON] Implement `count_distinct` and `sum_distinct` functions
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #38914:
URL: https://github.com/apache/spark/pull/38914#issuecomment-1340537857
merged into master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #38914: [SPARK-41381][CONNECT][PYTHON] Implement `count_distinct` and `sum_distinct` functions
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #38914:
URL: https://github.com/apache/spark/pull/38914#issuecomment-1337212273
```
DataFrame.columns values are different (66.66667 %)
[left]: Index(['a', 'count(DISTINCT b)', 'count(DISTINCT c)'], dtype='object')
[right]: Index(['a', 'count(b)', 'count(c)'], dtype='object')
```
seems a bug in existing DataFrame APIs, try to fix it in https://github.com/apache/spark/pull/38917
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #38914: [SPARK-41381][CONNECT][PYTHON] Implement `count_distinct` and `sum_distinct` functions
Posted by GitBox <gi...@apache.org>.
zhengruifeng closed pull request #38914: [SPARK-41381][CONNECT][PYTHON] Implement `count_distinct` and `sum_distinct` functions
URL: https://github.com/apache/spark/pull/38914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org