You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "cloud-fan (via GitHub)" <gi...@apache.org> on 2026/04/14 12:53:28 UTC
[PR] [SPARK-52818][SQL] Fix MergeSubplans creating nested WithCTE with cross-scope CTE references [spark]
cloud-fan opened a new pull request, #55338:
URL: https://github.com/apache/spark/pull/55338
### What changes were proposed in this pull request?
When a non-deterministic CTE (e.g. using `monotonically_increasing_id()`) is referenced in scalar subqueries and the result is displayed with `.show()` (which adds a `Limit`), `MergeSubplans` can create an outer `WithCTE` whose CTE defs reference CTE defs from an inner `WithCTE`. This causes `ReplaceCTERefWithRepartition` to crash with `NoSuchElementException` because it processes the outer CTE defs before the inner CTE defs have been added to the map.
The root cause: `MergeSubplans.apply` checks `case _: WithCTE => plan` to skip plans with CTEs, but this only matches when `WithCTE` is the **top-level** node. When `.show()` wraps the plan in `GlobalLimit(LocalLimit(WithCTE(...)))`, the top-level node is `GlobalLimit`, so `MergeSubplans` runs and creates a new outer `WithCTE` around the existing inner one — producing nested `WithCTE` nodes with cross-scope references.
**Repro:**
```sql
WITH cte_base AS (
SELECT recid, givenname, surname,
concat('t_', cast(monotonically_increasing_id() AS string)) AS __unique_id
FROM t
),
cte_tf_givenname AS (
SELECT givenname,
CAST(COUNT(*) AS DOUBLE) / (SELECT COUNT(givenname) FROM cte_base) AS tf
FROM cte_base WHERE givenname IS NOT NULL GROUP BY givenname
),
cte_tf_surname AS (
SELECT surname,
CAST(COUNT(*) AS DOUBLE) / (SELECT COUNT(surname) FROM cte_base) AS tf
FROM cte_base WHERE surname IS NOT NULL GROUP BY surname
)
SELECT cte_base.givenname, cte_base.surname,
cte_tf_givenname.tf, cte_tf_surname.tf
FROM cte_base
LEFT JOIN cte_tf_givenname ON cte_base.givenname = cte_tf_givenname.givenname
LEFT JOIN cte_tf_surname ON cte_base.surname = cte_tf_surname.surname
```
Calling `.show()` on this crashes. Calling `.collect()` works (no `Limit` wrapper).
The fix has two parts:
1. **`MergeSubplans`**: Change `case _: WithCTE => plan` to `case _ if plan.containsPattern(CTE) => plan` to skip plans that contain `WithCTE` anywhere in the tree, not just at the top level.
2. **`ReplaceCTERefWithRepartition`**: Add a defensive guard `if cteMap.contains(ref.cteId)` so that orphaned `CTERelationRef` nodes (if any) don't crash with `NoSuchElementException`.
### Why are the changes needed?
Bug fix. Queries with non-deterministic CTEs referenced in scalar subqueries crash with `java.util.NoSuchElementException: key not found` when using `.show()`.
### Does this PR introduce _any_ user-facing change?
Yes, queries that previously crashed now work correctly.
### How was this patch tested?
New tests added:
- `CTEInlineSuite`: end-to-end test with the exact repro query using `.show()`
- `InlineCTESuite`: unit test verifying `ReplaceCTERefWithRepartition` handles orphaned `CTERelationRef` gracefully
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52818][SQL] Fix MergeSubplans creating nested WithCTE with cross-scope CTE references [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #55338:
URL: https://github.com/apache/spark/pull/55338#issuecomment-4244153322
cc @peter-toth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52818][SQL] Fix MergeSubplans creating nested WithCTE with cross-scope CTE references [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #55338:
URL: https://github.com/apache/spark/pull/55338#issuecomment-4248528416
`SQLAppStatusListenerMemoryLeakSuite.no memory leak` is unrelated failure, thanks for the review, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52818][SQL] Fix MergeSubplans creating nested WithCTE with cross-scope CTE references [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #55338: [SPARK-52818][SQL] Fix MergeSubplans creating nested WithCTE with cross-scope CTE references
URL: https://github.com/apache/spark/pull/55338
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org