You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2026/10/20 01:59:30 UTC
[PR] branch-4.2 [fix](fe) Prune empty and cascading CTE plans (#62828, #61146) [doris]
englefly opened a new pull request, #69308:
URL: https://github.com/apache/doris/pull/69308
### What problem does this PR solve?
Issue Number: DORIS-29796
Related PR: #62828 (source of the pick), #61146 (source of the dependency fix), #68211
Problem Summary: backport the CTE empty/cascading pruning fix #62828 to branch-4.2, plus the
`ColumnPruning` CTE protection that #62828 implicitly depends on. Without it the three branch-4.2
RQG cases tracked by DORIS-29796 (6540158413 / DORIS-25384, 6540160970 / DORIS-25408,
6540160107 / DORIS-25409) keep failing.
#### Why #62828 alone is not enough on branch-4.2
`ColumnPruning` on branch-4.2 only protects `LogicalCTEProducer` with `skipPruneThis` and has no
`visitLogicalCTEAnchor`. When the required slots of an anchor do not include the producer outputs
(the normal case, because consumers reference the producer slots directly), the producer subtree is
pruned with an empty required set and `LogicalProject` materializes the empty projection as the
dummy `1 AS 1` projection. The rewrite plan therefore becomes
```
LogicalCteAnchor
|--LogicalProject ( projects=[1 AS 1] )
| +--LogicalCteProducer
+--...
```
which (a) breaks the `anchor.left() instanceof LogicalCTEProducer` invariant that #62828's
`normalizeCtePlan`/`refreshCteContext` relies on (`ClassCastException`), and (b) is an invalid CTE
structure: the materialized columns no longer match the slot mapping the consumers were built from,
which is what surfaces as `VSlotRef ... have invalid slot id` in case 6540160107.
Master has had the protection since #61146 (`visitLogicalCTEAnchor` +
`skipPruneThisAndFirstLevelChildren`); this PR ports exactly that hunk.
#### Hunk audit of the #62828 pick (source commit `c75eba5488f`)
| Source file | Disposition |
| --- | --- |
| `nereids/jobs/executor/Optimizer.java` | Ported verbatim (`normalizeCtePlan`, `refreshCteContext`, `containsCte`, `generateFilterPushedDownInlinedPlan`, method rename). |
| `nereids/rules/rewrite/CTEInliner.java` | Ported verbatim (`inlineByCurrentConsumerCount`, `InlineResult`, zero-consumer anchor removal, recursive anchor trimming). |
| `nereids/StatementContext.java` | Not applied: branch-4.2 already has `clearCteEnvironment()` from the multi-mode CTE inline work (#68211), and its variant also clears `consumerIdToLimitRows`. |
| `nereids/rules/rewrite/ClearContextStatus.java` | Not applied: 4.2 keeps the explicit per-map clears, which are a superset of what the picked code needs. |
| `qe/SessionVariable.java` | Not applied: 4.2 keeps `@VariableMgr.VarAttr(..., alias = "cbo_cte_inline_mode")`; the source commit only dropped the alias. |
| `.../CTEInlineTest.java` | Ported verbatim, including the new `refreshCteConsumersAfterNormalizeEliminatesEmptyBranch`. |
| `regression-test/suites/query_p0/cte/test_cbo_cte_inline_prune.groovy` | Ported verbatim, path adapted to the 4.2 layout (`suites/nereids_p0/cte/`; `suites/query_p0/cte` does not exist on 4.2). |
| `regression-test/data/query_p0/...` baselines | Paths adapted to the 4.2 layout (`data/nereids_p0/repeat/`, `data/nereids_rules_p0/unique_function/`); content unchanged. |
The dependency fix is a separate commit, ported from `5de1ed88d4b` (#61146) and limited to the
`ColumnPruning` hunk (12 added lines): CTE anchors and producers are no longer column pruned and
keep their first level children's full output requirement.
No BE changes.
### Release note
None
### Check List (For Author)
- Test
- [x] FE unit test: `./run-fe-ut.sh --run org.apache.doris.nereids.rules.rewrite.CTEInlineTest`
→ `Tests run: 2, Failures: 0, Errors: 0` (includes the new
`refreshCteConsumersAfterNormalizeEliminatesEmptyBranch`).
- [x] FE build: `./build.sh --fe` → `BUILD SUCCESS` on this branch (checkstyle included).
- [x] Original RQG cases replayed against a local 4.2 cluster (FE+BE built from this branch,
BE alive), using the full DDL/INSERT/session settings from the DORIS-29796 attachment
`repro-cte-empty-42.sql` (database renamed to a hyphen-free name so branch-4.2's parser
accepts it; no other change):
| case | expected | branch-4.2 before | this branch |
| --- | --- | --- | --- |
| 6540158413 | 0 rows | `size of sources must be greater than 0` | 0 rows, no error |
| 6540160970 | 0 rows | `size of sources must be greater than 0` | 0 rows, no error |
| 6540160107 | 0 rows | `VSlotRef col_int_undef_signed have invalid slot id: 15` | 0 rows, no error |
With only the #62828 commit replayed, cases 6540158413/6540160970 return 0 rows and
6540160107 still fails (`ClassCastException: LogicalProject cannot be cast to
LogicalCTEProducer`), which is why the `ColumnPruning` commit is part of this PR.
- [x] Candidate 6540166225 from the DORIS-29796 evidence could not be replayed: the evidence
attachment only carries its error text, not its SQL, so there is no input to verify locally.
- [x] Regression (local 4.2 cluster, JDBC `127.0.0.1:29731`):
- `-d nereids_p0/cte` → 9 suites, failed 0 (includes the new `test_cbo_cte_inline_prune`)
- `-s test_repeat_output_slot` → pass (baseline updated by the pick)
- `-s agg_with_unique_function` → pass (baseline updated by the pick)
- `-d nereids_rules_p0/cte` → 3 suites, failed 0
- `-d nereids_rules_p0/cte_limit_pushdown` → 1 suite, failed 0
- `-d nereids_rules_p0/column_pruning` → 9 suites, failed 0
- Introspective analysis
- [x] `normalizeCtePlan` iterates to a fixpoint instead of comparing `Plan.equals()`, because
`LogicalCTEAnchor`/`LogicalSubQueryAlias` ignore children in `equals()`.
- [x] Zero-consumer anchors are removed before memoization; anchors with an empty producer or
consumers below `inline_cte_referenced_threshold` are inlined.
- [x] CTE environment (`cteIdToProducer`, `cteIdToConsumers`, rewritten caches) is refreshed
from the normalized plan so memo construction does not see stale CTE state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.2 [fix](fe) Prune empty and cascading CTE plans (#62828, #61146) [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #69308:
URL: https://github.com/apache/doris/pull/69308#issuecomment-6282420540
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.2 [fix](fe) Prune empty and cascading CTE plans (#62828, #61146) [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #69308:
URL: https://github.com/apache/doris/pull/69308
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.2 [fix](fe) Prune empty and cascading CTE plans (#62828, #61146) [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69308:
URL: https://github.com/apache/doris/pull/69308#issuecomment-6281560302
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.2 [fix](fe) Prune empty and cascading CTE plans (#62828, #61146) [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #69308:
URL: https://github.com/apache/doris/pull/69308#issuecomment-6284368439
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.2 [fix](fe) Prune empty and cascading CTE plans (#62828, #61146) [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69308:
URL: https://github.com/apache/doris/pull/69308#issuecomment-6289099105
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.2 [fix](fe) Prune empty and cascading CTE plans (#62828, #61146) [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69308:
URL: https://github.com/apache/doris/pull/69308#issuecomment-6289098473
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.2 [fix](fe) Prune empty and cascading CTE plans (#62828, #61146) [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69308:
URL: https://github.com/apache/doris/pull/69308#issuecomment-6285593516
# FE Regression Coverage Report
Increment line coverage ` 93.10% (81/87)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69308_3b8f07eb43f4102f8701d120b635908b7547dd25_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69308_3b8f07eb43f4102f8701d120b635908b7547dd25_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.2 [fix](fe) Prune empty and cascading CTE plans (#62828, #61146) [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #69308:
URL: https://github.com/apache/doris/pull/69308#issuecomment-6285464981
skip buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org