You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "bhollis-dbx (via GitHub)" <gi...@apache.org> on 2026/09/16 22:31:36 UTC
[PR] [SPARK-59597][SQL] Avoid quadratic CTE attribute restoration [spark]
bhollis-dbx opened a new pull request, #58873:
URL: https://github.com/apache/spark/pull/58873
### What changes were proposed in this pull request?
Use `AttributeMap` when restoring attributes from CTE references to CTE definitions in `PushdownPredicatesAndPruneColumnsForCTEDef`. This replaces a linear semantic search per attribute with expression-ID lookup.
### Why are the changes needed?
The current implementation linearly searches the CTE attribute mapping for every restored attribute, making planning quadratic in CTE width. The cost reaches tens of milliseconds per rule application for wide CTEs and is repeated for CTE references and optimizer iterations.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Ran `PushdownPredicatesForCTEDefStalenessSuite`. All 7 tests passed.
Local benchmark output for the full rule:
| Columns | Before | After | Change |
|---:|---:|---:|---:|
| 1 | 2.38 us | 2.48 us | 4% slower |
| 2 | 2.85 us | 2.92 us | 2% slower |
| 5 | 5.02 us | 4.72 us | 6% faster |
| 10 | 8.15 us | 7.21 us | 12% faster |
| 20 | 14.78 us | 12.41 us | 16% faster |
| 50 | 41.36 us | 28.96 us | 30% faster |
| 100 | 100.11 us | 54.71 us | 45% faster |
| 500 | 1.65 ms | 0.27 ms | 6.1x faster |
| 1,000 | 6.53 ms | 0.58 ms | 11.2x faster |
| 2,000 | 26.54 ms | 1.33 ms | 19.9x faster |
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59597][SQL] Avoid quadratic CTE attribute restoration [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #58873:
URL: https://github.com/apache/spark/pull/58873#issuecomment-5897521755
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/186c4333e7f043937af31ab06a0134299d938598
- merged into branch-4.x https://github.com/apache/spark/commit/8d26b73dce726c3d86a1074e0ee0e0b7a0031a01
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59597][SQL] Avoid quadratic CTE attribute restoration [spark]
Posted by "n-young (via GitHub)" <gi...@apache.org>.
n-young commented on PR #58873:
URL: https://github.com/apache/spark/pull/58873#issuecomment-5896782709
LGTM!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59597][SQL] Avoid quadratic CTE attribute restoration [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b closed pull request #58873: [SPARK-59597][SQL] Avoid quadratic CTE attribute restoration
URL: https://github.com/apache/spark/pull/58873
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org