You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "peter-toth (via GitHub)" <gi...@apache.org> on 2026/07/25 17:18:26 UTC
[PR] [SPARK-58323][SQL][3.5] Materialize AliasAware output ordering and partitioning to avoid StackOverflowError [spark]
peter-toth opened a new pull request, #57533:
URL: https://github.com/apache/spark/pull/57533
### What changes were proposed in this pull request?
Backport of SPARK-58323 (#57502) to branch-3.5.
`AliasAwareQueryOutputOrdering.outputOrdering`, `PartitioningPreservingUnaryExecNode.outputPartitioning` and `BroadcastHashJoinExec`'s output-partitioning expansion build their results with `multiTransform`, which returns a `Stream` (Scala 2.12), and store the bounded result unforced. This forces them into strict collections with `.toList`, right after the existing `.take(...)` so the lazy short-circuit during candidate generation is preserved.
Tailored for branch-3.5 (differs from the master PR):
- Scala 2.12: the lazy collection here is `Stream`, not `LazyList` (its 2.13 successor).
- `PartitioningPreservingUnaryExecNode.outputPartitioning` IS fixed on 3.5, unlike the 4.0/4.1 backports: on Scala 2.12 the outer `Iterator.toSeq` there is lazy (a `Stream`), so the projected `PartitioningCollection` stored an unforced `Stream`. On Scala 2.13 the same `Iterator.toSeq` is strict (`List`), so 4.0/4.1 need no change there.
- The `BroadcastJoinSuite` single-element assertion change is omitted: on branch-3.5 `expandOutputPartitioning` always returns a `PartitioningCollection` (no `case p :: Nil` unwrap), so the single-element output shape is unchanged.
### Why are the changes needed?
Each plan node re-wraps the child ordering/partitioning's unforced `Stream`, so across a deep projection chain the deferred nesting grows unbounded and overflows the stack when the ordering/partitioning is later serialized or deeply traversed. On master this was observed as a driver `StackOverflowError` at `DAGScheduler.submitMissingTasks` during task serialization of a `SortedMergeCoalescedRDD` (SPARK-55715) on a large bucketed `MERGE`. That RDD and code path do not exist on branch-3.5, so this backport is defensive hardening of the same latent lazy-collection-in-output-ordering/partitioning hazard, which has existed since the `multiTransform` path was introduced in SPARK-42049.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New test in `ProjectedOrderingAndPartitioningSuite` asserting both `sameOrderExpressions` and the projected `PartitioningCollection.partitionings` are strict (not a `Stream`); it fails before this change and passes after. Existing `ProjectedOrderingAndPartitioningSuite` / `BroadcastJoinSuite` / `BroadcastJoinSuiteAE` pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58323][SQL][3.5] Materialize AliasAware output ordering and partitioning to avoid StackOverflowError [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #57533:
URL: https://github.com/apache/spark/pull/57533#issuecomment-5080285488
Ah yes, you are right, I forgot that. I can close this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58323][SQL][3.5] Materialize AliasAware output ordering and partitioning to avoid StackOverflowError [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57533:
URL: https://github.com/apache/spark/pull/57533#issuecomment-5080429609
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58323][SQL][3.5] Materialize AliasAware output ordering and partitioning to avoid StackOverflowError [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth closed pull request #57533: [SPARK-58323][SQL][3.5] Materialize AliasAware output ordering and partitioning to avoid StackOverflowError
URL: https://github.com/apache/spark/pull/57533
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org