You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "peter-toth (via GitHub)" <gi...@apache.org> on 2026/07/24 16:01:53 UTC
[PR] [SPARK-58324][SQL] Drop unused sameOrderExpressions from GroupPartitionsExec k-way merge ordering [spark]
peter-toth opened a new pull request, #57503:
URL: https://github.com/apache/spark/pull/57503
### What changes were proposed in this pull request?
`GroupPartitionsExec` builds a `SortedMergeCoalescedRDD` for the k-way merge and hands it a `LazyCodeGenOrdering` built from `child.outputOrdering`. The generated comparator (`GenerateOrdering`) only needs each `SortOrder`'s sort key (child, direction, null ordering), so this drops `sameOrderExpressions` -- planner-only metadata -- via a small `kWayMergeOrdering` helper before constructing the ordering, so it is not serialized with the RDD in every task.
### Why are the changes needed?
`sameOrderExpressions` is unused by the merge comparator and is unnecessary payload serialized with every task. It was also the vector for the `StackOverflowError` fixed in SPARK-58323 (an unforced, deeply-nested `LazyList`); not carrying it here removes this operator's exposure to any such ordering entirely (defense-in-depth), independent of that fix.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New unit test in `GroupPartitionsExecSuite` asserting `kWayMergeOrdering` keeps the sort key but drops `sameOrderExpressions`. Existing SPARK-55715 sorted-merge tests cover comparator correctness.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58324][SQL] Drop unused sameOrderExpressions from GroupPartitionsExec k-way merge ordering [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth closed pull request #57503: [SPARK-58324][SQL] Drop unused sameOrderExpressions from GroupPartitionsExec k-way merge ordering
URL: https://github.com/apache/spark/pull/57503
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58324][SQL] Drop unused sameOrderExpressions from GroupPartitionsExec k-way merge ordering [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #57503:
URL: https://github.com/apache/spark/pull/57503#issuecomment-5078040005
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/cce435599ed332cd5b785d49b26d4e7dde631d8c
- merged into branch-4.x https://github.com/apache/spark/commit/4a70dc07df39fd7d561e34750bd2d478b5665a17
- merged into branch-4.2 https://github.com/apache/spark/commit/fb34cbeeec8a041a6e8a96cc60e962e78116a19f
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58324][SQL] Drop unused sameOrderExpressions from GroupPartitionsExec k-way merge ordering [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #57503:
URL: https://github.com/apache/spark/pull/57503#issuecomment-5078041614
Thank you everyone for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org