You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/01 12:58:52 UTC
[GitHub] [spark] brandondahler opened a new pull request, #37361: [SPARK-39925][SQL] Add array_sort(column, comparator) overload to DataFrame operations
brandondahler opened a new pull request, #37361:
URL: https://github.com/apache/spark/pull/37361
### What changes were proposed in this pull request?
Adding a new `array_sort` overload to `org.apache.spark.sql.functions` that matches the new overload defined in [SPARK-29020](https://issues.apache.org/jira/browse/SPARK-29020) and added via #25728.
### Why are the changes needed?
Adds access to the new overload for users of the DataFrame API so that they don't need to use the `expr` escape hatch.
### Does this PR introduce _any_ user-facing change?
Yes, now allows users to optionally provide a comparator function to the `array_sort`, which opens up the ability to sort descending as well as sort items that aren't naturally orderable.
#### Example:
Old:
```
df.selectExpr("array_sort(a, (x, y) -> cardinality(x) - cardinality(y))");
```
Added:
```
df.select(array_sort(col("a"), (x, y) => size(x) - size(y)));
```
### How was this patch tested?
Unit tests update to validate that the overload matches the expression's behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #37361: [SPARK-39925][SQL] Add array_sort(column, comparator) overload to DataFrame operations
Posted by GitBox <gi...@apache.org>.
zero323 commented on PR #37361:
URL: https://github.com/apache/spark/pull/37361#issuecomment-1219803719
It seems like it has to be re-synced with upstream, to address black failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #37361: [SPARK-39925][SQL] Add array_sort(column, comparator) overload to DataFrame operations
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #37361:
URL: https://github.com/apache/spark/pull/37361#issuecomment-1201674401
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37361: [SPARK-39925][SQL] Add array_sort(column, comparator) overload to DataFrame operations
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37361:
URL: https://github.com/apache/spark/pull/37361#issuecomment-1221506404
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brandondahler commented on pull request #37361: [SPARK-39925][SQL] Add array_sort(column, comparator) overload to DataFrame operations
Posted by GitBox <gi...@apache.org>.
brandondahler commented on PR #37361:
URL: https://github.com/apache/spark/pull/37361#issuecomment-1218752851
> LGTM. are you also interested in adding this in SparkR and PySpark? We can do that in a separate PR.
I do think they should be added (I checked that they aren't already there), but I don't personally have availability to do so at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37361: [SPARK-39925][SQL] Add array_sort(column, comparator) overload to DataFrame operations
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37361:
URL: https://github.com/apache/spark/pull/37361#issuecomment-1218906004
Oops, it slipped through my fingers. Mind retriggering https://github.com/brandondahler/spark/runs/7585897593?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #37361: [SPARK-39925][SQL] Add array_sort(column, comparator) overload to DataFrame operations
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #37361: [SPARK-39925][SQL] Add array_sort(column, comparator) overload to DataFrame operations
URL: https://github.com/apache/spark/pull/37361
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #37361: [SPARK-39925][SQL] Add array_sort(column, comparator) overload to DataFrame operations
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #37361:
URL: https://github.com/apache/spark/pull/37361#issuecomment-1218970577
since [`pyspark/sql/tests/test_functions.py`](https://github.com/apache/spark/blob/master/python/pyspark/sql/tests/test_functions.py#L92-L103) will check the parity between PySpark and SQL, so I think we may need to add `array_sort` into `expected_missing_in_py`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brandondahler commented on pull request #37361: [SPARK-39925][SQL] Add array_sort(column, comparator) overload to DataFrame operations
Posted by GitBox <gi...@apache.org>.
brandondahler commented on PR #37361:
URL: https://github.com/apache/spark/pull/37361#issuecomment-1218916757
Clicked re-run all jobs on that linked run, let me know if there was something else you meant for me to do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37361: [SPARK-39925][SQL] Add array_sort(column, comparator) overload to DataFrame operations
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37361:
URL: https://github.com/apache/spark/pull/37361#issuecomment-1218906651
cc @zero323, @itholic, @zhengruifeng FYI (since we need to add PySpark and SparkR ones)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] brandondahler commented on pull request #37361: [SPARK-39925][SQL] Add array_sort(column, comparator) overload to DataFrame operations
Posted by GitBox <gi...@apache.org>.
brandondahler commented on PR #37361:
URL: https://github.com/apache/spark/pull/37361#issuecomment-1221422904
Rebased on lastest master changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org