You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Pajaraja (via GitHub)" <gi...@apache.org> on 2025/12/29 15:44:53 UTC
[PR] [SPARK-54864][SQL] Add rCTE nodes to NormalizePlan [spark]
Pajaraja opened a new pull request, #53636:
URL: https://github.com/apache/spark/pull/53636
### What changes were proposed in this pull request?
Added `UnionLoop` and `UnionLoopRef` to `NormalizePlan` and `CteIdNormalizer`.
### Why are the changes needed?
To add testing for recursive CTEs for single pass analyzer.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New tests in `NormalizePlanSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 4.5 via Cursor AI - Original draft of the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54864][SQL] Add rCTE nodes to NormalizePlan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #53636:
URL: https://github.com/apache/spark/pull/53636#issuecomment-3728061572
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54864][SQL] Add rCTE nodes to NormalizePlan [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #53636:
URL: https://github.com/apache/spark/pull/53636#issuecomment-3701964912
## JIRA Issue Information
=== Sub-task SPARK-54864 ===
Summary: Add plan normalization for recursive CTEs
Assignee: None
Status: Open
Affected: ["4.2.0"]
---
*This comment was automatically generated by GitHub Actions*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54864][SQL] Add rCTE nodes to NormalizePlan [spark]
Posted by "diffray-bot (via GitHub)" <gi...@apache.org>.
diffray-bot commented on PR #53636:
URL: https://github.com/apache/spark/pull/53636#issuecomment-3698102591
## Changes Summary
This PR adds support for normalizing UnionLoop and UnionLoopRef nodes in the plan comparison logic. It introduces new normalization methods in CteIdNormalizer and corresponding test cases to enable testing of recursive CTEs in the single-pass analyzer.
**Type:** feature
**Components Affected:** catalyst/plans/NormalizePlan, catalyst/plans/CteIdNormalizer, test suite for NormalizePlan
<details>
<summary><strong>Files Changed</strong></summary>
| File | Summary | Change | Impact |
|:-----|:--------|:------:|:------:|
| `...he/spark/sql/catalyst/plans/NormalizePlan.scala` | Added handling for UnionLoop and UnionLoopRef nodes in the plan normalization transformation and added corresponding normalization methods to CteIdNormalizer class. | ✏️ | 🟡 |
| `...ark/sql/catalyst/plans/NormalizePlanSuite.scala` | Added three new test cases for UnionLoopRef ID normalization, UnionLoop ID and outputAttrIds normalization, and complete recursive CTE normalization. | ✏️ | 🟡 |
</details>
<details>
<summary><strong>Architecture Impact</strong></summary>
- **New Patterns:** Parallel normalization patterns for recursive CTE nodes (similar to existing CTE handling)
- **Dependencies:** added imports: CTERelationDef, UnionLoop, UnionLoopRef in test file
- **Coupling:** Introduces tight coupling between UnionLoop/UnionLoopRef and CteIdNormalizer. The normalization logic assumes specific traversal order (UnionLoop before UnionLoopRef) similar to existing CTE handling.
</details>
**Risk Areas:** ID mapping correctness: The normalization relies on traversal order where UnionLoop is encountered before UnionLoopRef to build the mapping correctly, Duplicate ID handling in normalizeDef: The logic checks if a CTERelationDef ID is already mapped (new check at line 256), which could affect existing behavior for duplicate CTEs, UnionLoop ID mapping: The normalizeUnionLoop method only remaps if ID already exists, but doesn't insert new mappings. This is asymmetric compared to normalizeUnionLoopRef behavior
<details>
<summary><strong>Suggestions</strong></summary>
- Add documentation explaining the assumption about traversal order in plan transformation
- Consider adding a comment explaining why UnionLoop doesn't insert new mappings unlike UnionLoopRef
- Verify the logic in normalizeDef change (line 256-257) doesn't introduce unintended regressions for existing CTE handling
</details>
<sub>Full review in progress... | Powered by <a href="https://diffray.ai?utm_source=github-summary">diffray</a></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54864][SQL] Add rCTE nodes to NormalizePlan [spark]
Posted by "diffray-bot (via GitHub)" <gi...@apache.org>.
diffray-bot commented on PR #53636:
URL: https://github.com/apache/spark/pull/53636#issuecomment-3699348977
## Changes Summary
This PR adds support for normalizing recursive CTEs (rCTEs) in Spark's query plan normalization logic by introducing handlers for UnionLoop and UnionLoopRef nodes. This enables the single-pass analyzer to correctly compare semantically identical recursive CTE queries with different internal IDs.
**Type:** feature
**Components Affected:** Catalyst Query Plan Normalization, Recursive CTE Support, Query Plan Comparison Infrastructure
<details>
<summary><strong>Files Changed</strong></summary>
| File | Summary | Change | Impact |
|:-----|:--------|:------:|:------:|
| `...he/spark/sql/catalyst/plans/NormalizePlan.scala` | Added UnionLoop and UnionLoopRef normalization handlers, plus a fix to prevent duplicate CTE ID mappings | ✏️ | 🟡 |
| `...ark/sql/catalyst/plans/NormalizePlanSuite.scala` | Added three comprehensive test cases for UnionLoopRef, UnionLoop, and recursive CTE normalization | ✏️ | 🟢 |
</details>
<details>
<summary><strong>Architecture Impact</strong></summary>
- **New Patterns:** Visitor pattern extension for rCTE nodes, ID mapping pattern for maintaining referential integrity
- **Dependencies:** added: UnionLoop and UnionLoopRef logical plan classes
- **Coupling:** NormalizePlan normalization logic now explicitly depends on recursive CTE node types (UnionLoop, UnionLoopRef), increasing coupling to the CTE implementation details
</details>
**Risk Areas:** Bug fix in normalizeDef() changes behavior - need to verify it doesn't break existing non-recursive CTE normalization, ID remapping logic complexity - UnionLoopRef mapping uses counter differently than CTERelationRef, potential for confusion, Interaction between UnionLoop ID normalization and UnionLoopRef counter-based normalization - asymmetric pattern could be error-prone
<details>
<summary><strong>Suggestions</strong></summary>
- Consider adding comments explaining why UnionLoopRef uses counter-based assignment vs. mapping lookup (unlike UnionLoop)
- Verify the bug fix in normalizeDef() doesn't have unintended side effects on regular CTEs
- Consider adding integration tests that verify recursive CTEs normalize correctly within larger query plans
</details>
<sub>Full review in progress... | Powered by <a href="https://diffray.ai?utm_source=github-summary">diffray</a></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54864][SQL] Add rCTE nodes to NormalizePlan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #53636: [SPARK-54864][SQL] Add rCTE nodes to NormalizePlan
URL: https://github.com/apache/spark/pull/53636
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org