You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "li-boxuan (via GitHub)" <gi...@apache.org> on 2025/08/01 04:55:29 UTC
[PR] Test to showcase Aggregate followed by ORDER BY doesn't preserve orders [spark]
li-boxuan opened a new pull request, #51768:
URL: https://github.com/apache/spark/pull/51768
This is a demo of a potential Spark planner bug: it chooses HashAggregate over SortAggregate for a SQL query that requires ordering. I didn't find any official documentation that says "HashAggregate" is order-preserving, so Spark seems to choose a wrong plan (as shown below)
```
[info] List(HashAggregate(keys=[], functions=[first(key#1L, false)], output=[first(key)#3L])
[info] +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=33]
[info] +- *(2) HashAggregate(keys=[], functions=[partial_first(key#1L, false)], output=[first#6L, valueSet#7])
[info] +- *(2) Sort [key#1L ASC NULLS FIRST], true, 0
[info] +- Exchange rangepartitioning(key#1L ASC NULLS FIRST, 5), ENSURE_REQUIREMENTS, [plan_id=28]
[info] +- *(1) Project [id#0L AS key#1L]
[info] +- *(1) Range (0, 100, step=1, splits=2)
```
for a simple query
```sql
SELECT FIRST(val) FROM (SELECT val FROM t ORDER BY val)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53060] Test to showcase Aggregate followed by ORDER BY doesn't preserve orders [spark]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #51768:
URL: https://github.com/apache/spark/pull/51768#issuecomment-3142285503
`first()` is is non-deterministic:
https://github.com/apache/spark/blob/58e07e0f4cca1e3a6387a7e0c57faeb6c5ec9ef5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/First.scala#L49-L50
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53060] Test to showcase Aggregate followed by ORDER BY doesn't preserve orders [spark]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #51768:
URL: https://github.com/apache/spark/pull/51768#issuecomment-3143598375
In fact, `first_value` is an alias of `first`:
https://github.com/apache/spark/blob/c96b9710f50ed045d8e949db6c0e251bbd57b201/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L466-L467
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53060] Test to showcase Aggregate followed by ORDER BY doesn't preserve orders [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #51768: [SPARK-53060] Test to showcase Aggregate followed by ORDER BY doesn't preserve orders
URL: https://github.com/apache/spark/pull/51768
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53060] Test to showcase Aggregate followed by ORDER BY doesn't preserve orders [spark]
Posted by "li-boxuan (via GitHub)" <gi...@apache.org>.
li-boxuan commented on PR #51768:
URL: https://github.com/apache/spark/pull/51768#issuecomment-3142484845
> `first()` is is non-deterministic:
>
> https://github.com/apache/spark/blob/58e07e0f4cca1e3a6387a7e0c57faeb6c5ec9ef5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/First.scala#L49-L50
Thanks for engaging in this discussion. Although `first()` is documented to be non-deterministic: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.first.html, `first_value()` is not: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.first_value.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53060] Test to showcase Aggregate followed by ORDER BY doesn't preserve orders [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51768:
URL: https://github.com/apache/spark/pull/51768#issuecomment-3509014515
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org