You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/06/10 18:21:26 UTC
[GitHub] [spark] ueshin opened a new pull request, #36834: [SPARK-39419][SQL][3.3] Fix ArraySort to throw an exception when the comparator returns null
ueshin opened a new pull request, #36834:
URL: https://github.com/apache/spark/pull/36834
### What changes were proposed in this pull request?
Backport of #36812.
Fixes `ArraySort` to throw an exception when the comparator returns `null`.
Also updates the doc to follow the corrected behavior.
### Why are the changes needed?
When the comparator of `ArraySort` returns `null`, currently it handles it as `0` (equal).
According to the doc,
```
It returns -1, 0, or 1 as the first element is less than, equal to, or greater than
the second element. If the comparator function returns other
values (including null), the function will fail and raise an error.
```
It's fine to return non -1, 0, 1 integers to follow the Java convention (still need to update the doc, though), but it should throw an exception for `null` result.
### Does this PR introduce _any_ user-facing change?
Yes, if a user uses a comparator that returns `null`, it will throw an error after this PR.
The legacy flag `spark.sql.legacy.allowNullComparisonResultInArraySort` can be used to restore the legacy behavior that handles `null` as `0` (equal).
### How was this patch tested?
Added some tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #36834: [SPARK-39419][SQL][3.3] Fix ArraySort to throw an exception when the comparator returns null
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #36834:
URL: https://github.com/apache/spark/pull/36834#issuecomment-1152805039
Merged to branch-3.3. Thank you, @ueshin .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #36834: [SPARK-39419][SQL][3.3] Fix ArraySort to throw an exception when the comparator returns null
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #36834: [SPARK-39419][SQL][3.3] Fix ArraySort to throw an exception when the comparator returns null
URL: https://github.com/apache/spark/pull/36834
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org