You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2026/10/20 07:46:02 UTC
[PR] [fix](fe) Keep a renaming STRUCT cast instead of folding it into a no-op [doris]
englefly opened a new pull request, #69328:
URL: https://github.com/apache/doris/pull/69328
### What problem does this PR solve?
Issue Number: DORIS-29733, DORIS-29734
Related PR: None
Problem Summary:
A STRUCT -> STRUCT cast converts positionally and its result carries the target field names, but
the FE treated a cast that only renames fields as a no-op and dropped the CAST node when
serializing expressions to thrift: `CastExpr.noOp` comes from `Type.matchExactType`, whose struct
branch compared the fields positionally and ignored their names. The folded-away cast also
overwrote the child expression's type with the target type, so the FE bookkeeping and the physical
column disagreed: the value kept the source field names, while the ELEMENT_AT binding and the
nested column pruning of the cast target were planned against the target names.
Reproduction with the default session variables on a single FE/BE:
```sql
CREATE TABLE t (id INT, s STRUCT<a:STRING,b:STRING>) ENGINE = OLAP
DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t VALUES (1, named_struct('a', 'left', 'b', 'right'));
SELECT element_at(CAST(s AS STRUCT<b:STRING,a:STRING>), 'b') FROM t;
```
Before the fix:
* with nested column pruning on (the default) the pruned slot kept only source field `a`, and the
query failed with
`errCode = 2, detailMessage = ... [RUNTIME_ERROR]Element not found for function element_at: name b not found in Struct(a:Nullable(String))`;
* with nested column pruning off the query silently returned `right` (the source field `b`)
instead of `left`;
* reading both target fields returned (`right`, `left`) instead of (`left`, `right`), while the
whole value CAST correctly returned `{b:'left', a:'right'}`.
`EXPLAIN VERBOSE` is correct in every case because the Nereids plan keeps the cast; only the thrift
sent to BE lost it, which is what makes this easy to misdiagnose as a pruning bug.
Root cause in one line: `noOp == true` meant "no CAST_EXPR node", and BE's `element_at` resolves the
field name against the type attached to the argument column in the Block (`try_get_position_by_name`
on the struct type of the block position). With the cast gone, that type was the scan type, i.e.
the source field names. BE has an explicit hook for this FE/BE type divergence
(`VExpr::execute_type`, "for slot-ref expressions the type of the returned Column may not match
data_type"), which is exactly the gap the elided cast fell into.
Fix:
A struct field name is part of the value identity, so `Type.matchExactType` now also requires the
field names to match (comparing the already normalized lower case name, consistent with BE's case
insensitive struct field lookup). A rename-only cast is serialized instead of being skipped, and BE
relabels the value with the target type. Since the positional field types are unchanged, BE keeps
the identity cast wrapper (`prepare_impl` compares with `DataTypeStruct::equals`, which ignores
names), so no data is copied. Renames nested in ARRAY/MAP are covered by the same recursion, and
`try_cast` inherits the fixed no-op decision.
After the fix both toggle states return the positional result, b=`left` and a=`right`, matching the
whole value CAST, and per-field pruning is preserved: the scan still reads only the source field
behind the accessed target field (`all access paths: [s.a]` / `[s.b]`).
### Release note
Fixed a wrong result and an "Element not found for function element_at" error when a
STRUCT -> STRUCT cast renames fields and its result is accessed by field name.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. A cast that only renames STRUCT fields is no longer folded away, so its result
carries the target field names as the type system already promised. Casts whose field names
already match keep the existing no-op behavior.
- Does this need documentation?
- [x] No.
Test detail:
* Unit test: `mvn test -pl fe-core -am -Dtest='CastExprHashPartitionTest,CastMonotonicityTest,CastTest,CheckCastTest,SimplifyCastRuleTest,TryCastTest,TypeTest,ExprToThriftBehaviorTest'` passes.
`ExprToThriftBehaviorTest#testStructRenameCastIsNotNoOp` asserts `isNoOp() == false` and that the
serialized tree is `CAST_EXPR(type=struct<b,a>)` + `SLOT_REF(type=struct<a,b>)`;
`#testStructRenameNestedInArrayCastIsNotNoOp` covers the recursion;
`#testSameStructTypeCastIsStillNoOp` pins that the no-op optimization is kept;
`TypeTest#testStructWithNestedFieldsExactMatch` covers the match rule itself.
* Regression test: `./run-regression-test.sh --run -d nereids_rules_p0/column_pruning` — 16 suites
pass, including the new `struct_cast_rename_pruning`, which asserts the values and
`all access paths: [s.a]` / `[s.b]` for pruning on and off.
* Manual test: the snippet above plus WHERE / GROUP BY / ORDER BY / subquery / CTE / `INSERT INTO` a
table whose struct column is named differently / ARRAY and MAP nested renames, all with nested
column pruning both on and off, on a local FE built from this commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep a renaming STRUCT cast instead of folding it into a no-op [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69328:
URL: https://github.com/apache/doris/pull/69328#issuecomment-6307563193
<details>
<summary>TPC-H: <b>Total hot run time: 28748 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3213ff0fa2fa8c7a0838b16237ea152486993029, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17594 5214 5055 5055
q2 2138 331 263 263
q3 10198 1398 819 819
q4 4677 457 340 340
q5 7423 796 519 519
q6 180 180 148 148
q7 738 791 608 608
q8 9247 1342 1124 1124
q9 5546 4207 4204 4204
q10 6814 1300 1013 1013
q11 426 244 220 220
q12 638 396 273 273
q13 18123 3896 3027 3027
q14 283 259 244 244
q15 q16 741 713 667 667
q17 1208 761 599 599
q18 7026 6188 6123 6123
q19 1112 959 584 584
q20 405 330 220 220
q21 5429 2437 2463 2437
q22 376 328 261 261
Total cold run time: 100322 ms
Total hot run time: 28748 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6010 5928 6127 5928
q2 746 606 628 606
q3 5634 5755 5240 5240
q4 2007 2179 1566 1566
q5 5626 5832 5646 5646
q6 248 185 140 140
q7 2227 2012 1848 1848
q8 3049 2661 2701 2661
q9 7986 7827 7634 7634
q10 3637 3569 3168 3168
q11 598 407 382 382
q12 664 674 497 497
q13 3296 3580 3054 3054
q14 294 285 259 259
q15 q16 656 683 633 633
q17 7757 7021 6937 6937
q18 13065 12166 12818 12166
q19 900 792 781 781
q20 2134 2164 1942 1942
q21 5604 4590 4666 4590
q22 495 472 401 401
Total cold run time: 72633 ms
Total hot run time: 66079 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep a renaming STRUCT cast instead of folding it into a no-op [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69328:
URL: https://github.com/apache/doris/pull/69328#issuecomment-6332918410
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep a renaming STRUCT cast instead of folding it into a no-op [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #69328:
URL: https://github.com/apache/doris/pull/69328#issuecomment-6306368101
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep a renaming STRUCT cast instead of folding it into a no-op [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly merged PR #69328:
URL: https://github.com/apache/doris/pull/69328
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep a renaming STRUCT cast instead of folding it into a no-op [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69328:
URL: https://github.com/apache/doris/pull/69328#issuecomment-6313845029
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/1360)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69328_3213ff0fa2fa8c7a0838b16237ea152486993029_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69328_3213ff0fa2fa8c7a0838b16237ea152486993029_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep a renaming STRUCT cast instead of folding it into a no-op [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69328:
URL: https://github.com/apache/doris/pull/69328#issuecomment-6307680215
<details>
<summary>TPC-DS: <b>Total hot run time: 150355 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3213ff0fa2fa8c7a0838b16237ea152486993029, data reload: false
query5 4307 597 444 444
query6 437 199 192 192
query7 4799 433 215 215
query8 319 159 170 159
query9 8766 3065 3107 3065
query10 478 315 251 251
query11 6056 3876 3548 3548
query12 153 90 87 87
query13 1238 429 318 318
query14 6035 3864 3632 3632
query14_1 3395 3382 3400 3382
query15 204 203 196 196
query16 946 422 395 395
query17 828 655 513 513
query18 2413 413 294 294
query19 197 177 129 129
query20 85 85 79 79
query21 210 121 100 100
query22 12945 13019 12987 12987
query23 13507 12793 12409 12409
query23_1 12564 12513 12456 12456
query24 7408 818 454 454
query24_1 466 459 458 458
query25 505 395 327 327
query26 1240 252 133 133
query27 2815 424 264 264
query28 4668 1904 1902 1902
query29 1538 576 419 419
query30 303 217 186 186
query31 1004 932 808 808
query32 181 96 85 85
query33 516 297 238 238
query34 985 849 495 495
query35 734 777 657 657
query36 852 814 744 744
query37 123 100 83 83
query38 1775 1738 1666 1666
query39 799 810 747 747
query39_1 746 730 742 730
query40 217 119 99 99
query41 81 85 85 85
query42 86 85 84 84
query43 350 362 312 312
query44 1288 675 678 675
query45 178 191 164 164
query46 810 959 563 563
query47 2938 2969 2827 2827
query48 296 306 222 222
query49 602 401 302 302
query50 661 308 199 199
query51 10498 10219 10185 10185
query52 74 78 70 70
query53 181 201 151 151
query54 221 209 172 172
query55 73 66 61 61
query56 244 205 214 205
query57 1652 1629 1555 1555
query58 279 260 250 250
query59 2135 2124 1952 1952
query60 287 223 224 223
query61 145 149 145 145
query62 373 325 298 298
query63 188 149 153 149
query64 2734 947 756 756
query65 3310 3280 3256 3256
query66 1776 479 291 291
query67 20113 20299 20090 20090
query68 3297 971 608 608
query69 395 280 249 249
query70 811 774 790 774
query71 305 237 220 220
query72 2792 2617 2322 2322
query73 520 516 291 291
query74 4634 4520 4366 4366
query75 2357 2287 1897 1897
query76 2392 1017 576 576
query77 342 380 270 270
query78 9664 9669 9181 9181
query79 864 849 510 510
query80 460 403 330 330
query81 516 313 272 272
query82 197 131 103 103
query83 186 188 175 175
query84 288 122 100 100
query85 758 413 371 371
query86 290 232 230 230
query87 2002 1965 1811 1811
query88 3541 2648 2626 2626
query89 312 279 241 241
query90 2099 184 165 165
query91 158 153 135 135
query92 94 101 85 85
query93 963 972 567 567
query94 455 307 267 267
query95 563 393 312 312
query96 660 477 234 234
query97 2427 2443 2308 2308
query98 159 153 151 151
query99 751 765 641 641
Total cold run time: 233345 ms
Total hot run time: 150355 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep a renaming STRUCT cast instead of folding it into a no-op [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #69328:
URL: https://github.com/apache/doris/pull/69328#issuecomment-6309810974
run nonConcurrent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep a renaming STRUCT cast instead of folding it into a no-op [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69328:
URL: https://github.com/apache/doris/pull/69328#issuecomment-6332917937
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep a renaming STRUCT cast instead of folding it into a no-op [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69328:
URL: https://github.com/apache/doris/pull/69328#issuecomment-6307742751
<details>
<summary>ClickBench: <b>Total hot run time: 24.29 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3213ff0fa2fa8c7a0838b16237ea152486993029, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.26 0.09 0.09
query4 1.61 0.10 0.10
query5 0.23 0.24 0.24
query6 1.18 0.69 0.67
query7 0.03 0.01 0.00
query8 0.05 0.04 0.04
query9 0.40 0.36 0.35
query10 0.56 0.55 0.60
query11 0.19 0.14 0.14
query12 0.19 0.15 0.14
query13 0.48 0.49 0.47
query14 1.04 1.02 1.01
query15 0.64 0.60 0.61
query16 0.35 0.36 0.31
query17 1.14 1.17 1.11
query18 0.20 0.18 0.19
query19 2.11 2.00 1.96
query20 0.02 0.01 0.01
query21 15.42 0.20 0.13
query22 4.89 0.05 0.06
query23 16.41 0.32 0.13
query24 2.83 0.63 0.42
query25 0.12 0.06 0.05
query26 0.89 0.21 0.17
query27 0.05 0.05 0.05
query28 3.03 0.70 0.38
query29 12.50 4.15 3.26
query30 0.29 0.13 0.10
query31 2.77 0.56 0.31
query32 3.22 0.59 0.50
query33 3.10 3.30 3.22
query34 15.34 3.96 3.37
query35 3.37 3.33 3.39
query36 0.54 0.44 0.43
query37 0.10 0.06 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.18 0.15 0.14
query41 0.09 0.04 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 96.1 s
Total hot run time: 24.29 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep a renaming STRUCT cast instead of folding it into a no-op [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69328:
URL: https://github.com/apache/doris/pull/69328#issuecomment-6286037460
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep a renaming STRUCT cast instead of folding it into a no-op [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #69328:
URL: https://github.com/apache/doris/pull/69328#issuecomment-6306365747
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Keep a renaming STRUCT cast instead of folding it into a no-op [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69328:
URL: https://github.com/apache/doris/pull/69328#issuecomment-6309990032
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/1273)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69328_3213ff0fa2fa8c7a0838b16237ea152486993029_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69328_3213ff0fa2fa8c7a0838b16237ea152486993029_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org